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Abstract 

The  purpose  of  this  study  is  to  present  a  performance 
evaluation  system  based  upon  benchmarking  techniques  as  an 
alternative  method  of  assessing  performance  within 
operational  contracting  squadrons. 

The  research  focused  upon  four  main  objectives:  (1) 
determine  all  the  performance  factors  that  currently  exist 
within  a  typical  contracting  squadron;  (2)  determine  which 
of  these  factors  are  considered  significant  for  measuring 
organizational  performance;  (3)  determine  whether  a 
benchmark  can  be  established  for  each  of  these  factors;  and 
(4)  determine  whether  a  viable  performance  evaluation  system 
can  be  leveloped  based  upon  benchmarking  techniques. 

To  accomplish  these  objectives,  an  extensive  review  of 
literature  pertaining  to  performance  evaluation  systems  was 
conducted.  This  review  centered  on  the  characteristics  of 
effective  performance  evaluation  systems.  These 
characteristics  were  compared  to  the  current  inspection 
system  employed  by  the  Air  Force  and  ihconsistencies  were 
identified.  Using  this  information,  benchmarking  techniques 
were  applied  in  an  effort  to  incorporate  the  characteristics 
of  an  effective  contracting  performance  evaluation  system. 
Recommendations  were  offered  to  the  Air  Force  contracting 
community  and  suggestions  for  future  research  were  given. 


TOWARD  ALTERNATIVE  METRICS  FOR  MEASURING  PERFORMANCE  WITHIN 
OPERATIONAL  CONTRACTING  SQUADRONS: 

AN  APPLICATION  OF  BENCHMARKING  TECHNIQUES 

I- 

Overview 

For  many  years /  organizational  managers  have  aspired  to 
achieve  optimal  performance  within  their  organizations. 

This  pursuit  of  excellence  defines  a  manager's  role  within 
an  organization  and  presents  a  daunting  challenge  for  these 
select  individuals.  To  achieve  optimal  performance,  a 
manager  must  strive  for  increased  organizational 
effectiveness.  Organizational  effectiveness  itself  is  a 
fundamental  element  of  an  organization's  ability  to 
competently  operate  and  succeed  in  today ' s  working 
environment.  Albanese  refers  to  organizational 
effectiveness  as  "the  bottom  line,  .  .  .  the  reason  for 
memagerial  work."  (Albanese,  1975;ix).  Further 
understanding  of  the  term  organizational  effectiveness 
requires  an  accepted  definition  of  effectiveness.  "The  term 
[organizational]  effectiveness  relates  to  the  accomplishment 
of  an  organization's  goals  and  objectives.  An  organization 
is  effective  when  its  goals  are  accomplished;  it  is 
ineffective  when  they  are  not"  (Herbert,  1989:588). 
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There  is  no  question  that  organizations  strive  to 

achieve  optimal  effectiveness/  but  all  organizations  vary  in 

ability  and  performance  level.  To  adequately  assess 

organizational  performance/  an  effective  performance 

evaluation  system  must  be  used.  Herbert  states: 

management  control  involves  the  measurement  and 
evaluation  of  program  activities  to  determine  if 
policies  and  objectives  are  being  accomplished  as 
efficiently  eind  effectively  as  possible.  .  . 

Memagement  control  provides  the  basic  structure  for 
coordinating  the  day-to-day  activities  involved  in 
ensuring  that  the  organization's  resources  are 
appropriately  used  in  the  pursuit  of  goals  rnd 
objectives.  (Hebert/  1989:591) 

Many  different  types  of  performance  evaluation  systems 
exist.  The  challenge  to  the  organizational  manager  is 
determining  and  implementing  a  system  that  will  effectively 
assess  organizational  performance. 

To  address  this  challenge/  this  study  will  review  and 
analyze  both  industry  and  government  organizational 
performance  evaluation  systems  and  apply  the  findings  to  the 
development  of  an  alternative  evaluation  system  that  is 
capable  of  measuring  performance  within  a  specific  function 
of  the  Department  rf  Defense  (DOD)— Air  Force  operational 
contracting  squadrons.  The  benefits  associated  with  this 
alternative  evaluation  system  may  well  result  in  improved 
organizational  performance. 

Chapter  I  will  provide  an  introduction  to  the  general 
issue,  research  problem,  and  investigative  questions. 
Further,  the  chapter  will  address  definitions  associated 
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with  the  thesis  effort,  in  addition  to  the  scope  of  the 
research.  Finally,  the  chapter  will  discuss  the  mission  and 
structure  of  an  operational  contracting  squadron,  as  well  as 
the  performance  evaluation  system  currently  used  to  assess 
operational  contracting  squadron  performance. 

General  Issue 

In  order  for  the  military  services  to  successfully 
operate  in  today's  changing  environment,  the  DOi),  and  in 
tarn  the  Air  Force,  have  identified  a  need  to  reorganize  and 
downsize.  At  the  same  time,  all  military  organizations  are 
challenged  with  the  task  of  increasing  their  overall  levels 
of  efficiency  and  effectiveness  (Air  Force  Policy  Letter, 
1990:1).  In  order  to  adapt  to  this  changing  environment, 
the  Air  Force  has  embarked  upon  a  journey  toward  a  quality 
revolution.  This  revolution  requires  a  culture  change  and  a 
competitive,  customer-focused  workforce  whose  goal  is 
continuous  process  improvement  (Alvarado,  1993:8). 

Since  1992,  the  Air  Force  has  been  moving  toward  the 
concept  of  a  Quality  Aix  Force  (QAF)  which  is  directed  at 
continv  usly  improving  quality  and  productivity  by  use  of  a 
variety  of  time-proven  management  methods  and  numerous 
statistical  and  analytical  tools  (Bossert,  1993:16).  Within 
this  movement,  one  of  the  major  taskings  organizational 
managers  are  facing  is  the  development  of  an  adequate 
performance  evaluation  system  capable  of  assessing 


organizational  performance  (Danis,  1993:4).  Since  the 
development  of  the  current  Air  Force  performance  evaluation 
system  occurred  prior  to  the  implementation  of  the  QAF 
concept,  it  appears  the  current  system  fails  to  incorporate 
many  of  the  principles  of  this  new  concept. 

Realizing  this  inadequacy.  Colonel  Robert  Cordano, 
Chief  of  Contracting,  Air  Combat  Command  (ACC),  has 
challenged  all  ACC  contracting  personnel  to  find  an 
alternative  performance  evaluation  system  capable  of 
assessing  contracting  squadron  performance  in  this  new 
environment.  In  his  letter  dated  8  October  1992,  Colonel 
Cordano  states  that  a  "beneficial  thesis  effort  could  be 
directed  at  estedslishing  a  metric  or  metrics  to  determine 
the  effectiveness  of  a  given  contracting  squadron's  support 
to  the  wing's  mission  needs."  This  letter  is  included  as 
Appendix  A  to  this  thesis. 

Pgqbigm  Slatsmgnt. 

With  the  changing  acquisition  environment  and  the  push 
toward  a  Quality  Air  Force,  a  metric  to  assess  overall 
effectiveness  within  contracting  squadrons  must  be 
developed.  The  purpose  of  this  thesis  is  to  address  this 
need  by  presenting  an  alternative  performance  evaluation 
system  capable  of  assessing  organizational  performance 
within  this  new  acquisition  environment. 
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Investigative  Questions 


Four  investigative  questions  will  guide  the  thesis 
ei^fort.  The  four  questions  are: 

1.  What  performance  factors  currently  exist  within  a 
typical  contracting  squadron? 

2.  Which  of  these  performance  factors  are  considered 
significant  for  measuring  organizational  performance  and 
therefore  should  be  included  in  the  development  of  an 
effective  performance  evaluation  system? 

3.  Can  a  benchmark  be  established  for  each  significant 
performance  factor? 

4.  Can  a  viable  performance  evaluation  system,  capable  of 
accurately  measuring  overall  effectiveness  for  a  typical 
contracting  squadron,  be  developed  using  benchmarking 
techniques  and  concepts? 


Scope  of  the  Study 

The  scope  of  this  study  is  limited  to: 

1.  Air  Force  Operational  Contracting  Squadrons  within 
the  Continental  United  States  (CONUS)  in  a  peacetime 
environment . 

2.  The  perceptions  and  attitudes  of  Squadron 
Commanders,  Deputy  Base  Contracting  Officers  (DBCO) ,  and 
Executive  Noncommissioned  Officers  (NCO)  within  each 
operational  contracting  squadron. 

Contracting  Squadron  Responsibilities  and  Structure 

Contracting  squadrons  are  operational  units  located  at 
base  installations  throughout  the  CONUS  and  overseas. 
Currently,  119  squadrons  exist  within  the  Air  Force,  86 
within  the  CONUS  (Directory  of  Air  Force  Operational 
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Contracting  Activities,  1993 :A1-A12 ) .  The  primary 
responsibilities  of  contracting  squadrons  are  to  solicit, 
review,  award,  and  administer  all  contracting  actions  for  an 
operational  base,  as  well  as  analyze  and  coordinate  all 
contracting  data  and  contracting  matters  that  may  affect 
base  operations  (AFR  70-8,  1979:1). 

Air  Force  Regulation  26-1  states  that  the  typical 
contracting  squadron  is  assigned  a  Squadron  Commander, 

Deputy  Base  Contracting  Officer,  and  Executive  NCO. 

Together  with  administrative  support  personnel,  these 
individuals  comprise  the  senior  management  element  of  the 
operational  contracting  squadron.  These  individuals  oversee 
the  daily  operation  of  four  flights.  The  four  flights  are 
Construction,  Services,  Commodities,  and  Management  and 
Analysis  Support . 

The  first  flight,  Construction,  is  responsible  for 
purchasing  and  administering  all  construction  projects,  as 
well  as  providing  architect-engineering  support  design  for 
certain  projects.  Construction  projects  are  normally 
limited  to  alteration,  repair,  or  maintenance  of  existing 
facilities.  Examples  of  construction  contracts  include  the 
repair  or  replacement  of  built-up  roofs,  renovation  and 
refurbishment  of  existing  structures,  and  the  removal  of 
underground  fuel  storage  tanks. 

The  Services  Flight  is  responsible  for  all  contracts 
that  provide  service  support  for  base  operations.  Examples 
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of  service  contracts  include  vehicle  rental^  laundry  and  dry 
cleaning,  military  family  housing  maintenance,  as  well  as 
purchase  and  administration  of  many  smaller  service 
requirements . 

The  third  flight,  Commodities,  is  responsible  for 
purchasing  and  administering  a  variety  of  commodities  and 
equipment  contracts.  Examples  of  commodities  contracts 
include  the  purchase  of  typewriter  ribbons,  storage 
buildings,  and  medical  supplies. 

The  final  flight.  Management  Analysis  and  Support,  is 
not  responsible  for  purchasing  and  administering  contracts. 
Instead,  this  flight  is  responsible  for  operating  and 
maintaining  the  Base  Contracting  Automated  System  (BCAS) 
which  provides  data  and  information  with  respect  to  overall 
squadron  performance.  Data  and  information  is  provided  to 
the  senior  management  element  in  the  form  of  monthly 
computer  reports. 

Current  Evaluation  Procedures 

In  order  for  an  organization  to  assess  performance,  an 
evaluation  system  must  exist  that  provides  performance 
feedback  to  the  organization.  The  current  method  of 
evaluation  for  all  Air  Force  organizations  is  the  Air  Force 
Inspection  System,  This  system  is  used  by  the  Inspector 
General  (IG)  when  conducting  a  Unit  Effectiveness  Inspection 
(UEI)  of  contracting  squadrons.  As  stated  in  AFR  123-1,  Air 


7 


Force  Inspection  System,  the  specific  purposes  of  the 
inspection  system  are: 

(1)  Measure  US  Air  Force  readiness,  as  shown  in 
operational  unit  and  system  discipline  and  perfonaance. 

(2)  Measure  the  effectiveness  and  efficiency  of 
units,  functions,  programs,  and  guidance. 

{3)  Evaluate  Air  Force  and  unit  internal  controls  and 
identify  good  management  methods  for  crossfeed  to 
other  units . 

(4)  Help  define  command  priorities  to  units  to 
perform  their  designated  missions. 

(5)  Validate  and  compare  management  information 
reported  through  other  staff  agencies. 

(6)  Follow  up  on  corrective  actions  taken  on 
previously  identified  deficiencies. 

(7)  Detect  and  prevent  fraud,  waste,  and  abuse. 

(8)  Identify  and  resolve  readiness  issues  and  other 
important  problems  (Afr  123-1,  1989:5). 

This  particular  evaluation  system  is  conducted  by  the 

IG  to  evaluate  the  overall  performance  of  an  operational 

contracting  squadron.  The  actual  checklists  and  inspection 

guides  implemented  throughout  the  inspection  are  based  upon 

performance  factors,  or  indicators,  reported  and  tracked 

through  BCAS  reports. 


wj-vii-p 

1.  ACC  — ■  Air  Combat  Command 

2.  AFMC  —  Air  Force  Materiel  Conanand 

3 .  Air  Force  Inspection  System  —  method  of  assessing 
the  efficiency  of  management  as  well  as  the  effectiveness 
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and  economy  of  operations,  usually  accomplished  by  the  Air 
Force-  Inspector  General  during  a  Unit  Effectiveness 
Inspection. 

4.  AMC  —  Air  Mobility  Command 

5.  Aye  —  Air  Training  Command 

6.  BCAS  report  —  Base  Contracting  Automated  System 
repox't.  Monthly,  quarterly,  and  yearly  report  that  tracks 
all  contracting  actions  and  information  in  order  to  provide 
performance  information. 

7.  Benchmark  —  means  to  evaluate  performance  between 
similar  organizations; . 

8.  Contract  Administration  Lead  Time  (CALT)  —  the 
elapsed  time  between  receipt  and  fulfillment  of  a  purchase 
request . 

9.  Deputy  Base  Contracting  Officer  fPBCQ^  —  the 
senior  ranking  civilian  within  the  operational  contracting 
squadron,  subordinate  only  to  the  squadron  commander.  Focal 
point  for  all  civilian  matters  as  well  as  numerous 
contracting  actions. 

10.  Executive  NCO  — ■  usually  the  senior  ranking 
enlisted  member  within  the  squadron;  almost  always  the  most 
ixperienced  and  knowledgeable  individual  among  enlisted 
members  within  the  squadron. 

11.  Orqaniz^ational  Effectiveness  —  the  degree  to 
which  an  organization  satisfies  or  achieves  its  goals, 
objectives,  or  mission. 
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12.  Pexforwance  Evaluation  System  —  a  system,  based 
upon  accurate  and  meaningful  performance  measures,  capable 
of  evaluating  organizational  performance  with  respect  to 
overall  effectiveness. 

,  13.  Performance  Measure  —  factors,  or  indicators,  of 

work  activity  found  within  an  organization,  capable  of  being 
used  in  a  performance  evaluation  system  to  evaluate  overall 
organizational  effectiveness . 

14.  Senior  Management  Element  —  senior  management  of 
an  operational  contracting  squadron;  usually  comprised  of 
Contracting  Squadron  Commander,  Deputy  Base  Contracting 
Officer,  and  Executive  NCO. 

15.  Squadron  Commander/Base  Contracting  Officer  —  the 
senior  ranking  military  officer  within  am  operational 
contracting  squadron  with  decision  making  authority  for  all 
squadron  matters  as  well  as  numerous  contracting  matters. 

16.  VJlit,  E££ag.tiyei[\§B.g.  Ins.p.g?tlpn  —  Air  Force 
inspection  of  operational  units  within  the  Air  Force, 
conducted  by  the  Inspector  General  every  12  to  18  months. 

g-wnmary 

The  environment  in  which  the  military  is  presently 
operating  has  changed  dramatically  over  the  past  several 
years.  Force  reduction  and  reorganization,  as  well  as  the 
move  toward  total  quality,  has  impacted  the  manner  in  which 
all  military  organizations  now  perform  their  mission. 
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Operational  contracting  squadrons  are  no  different.  Even  in 
a  changed  environment,  organizational  effectiveness  remains 
one  of  the  primary  concerns  for  senior  management. 

The  manner  in  which  operational  contracting  squadrons 
now  perform  their  mission  has  created  a  need  to  review  and 
update  the  way  the  squadron  measures  and  evaluates 
performance.  The  system  used  to  evaluate  contracting 
squadron  performance  was  developed  and  implemented  prior  to 
the  move  toward  total  quality.  In  an  effort  to  improve 
quality,  this  thesis  will  address  those  areas  concerned  with 
the  development  of  an  alternative  performance  evaluation 
system,  such  that  the  proposed  system  is  capable  of 
measuring  performance  for  an  operational  contracting 
squadron. 
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II.  Lltfitatuia  Sgziaw 


overview 

The  concept  o£  organizational  effectiveness  implies 
different  things  to  different  individuals  and  can  be 
assessed  using  diverse  evaluation  systems.  These  disparate 
evaluation  systems  possess  both  advantages  and  disadvantages 
and  must  be  reviewed  by  the  organizational  manager  to 
determine  which  one  allows  for  accurate  and  meaningful 
assessment  of  the  organization’s  performance. 

This  chapter  will  explore  the  concept  of  organizational 
effectiveness  and  the  many  types  of  performance  evaluation 
systems  used  to  assess  performance^  as  well  as  the  benefits 
of  evaluating  performance.  Based  upon  this  review,  the 
authors  will  then  decipher  the  applicable  concepts  that  will 
be  applied  in  this  research.  Next,  the  chapter  will  focus 
on  the  Air  Force  evaluation  system  along  with  the  perceived 
limitations  associated  with  this  system.  Finally,  the 
chapter  will  conclude  with  a  description  of  benchmarking  and 

its  potential  for  satisfying  the  limitations  identified  with 

1 

the  current  Air  Force  system. 

Qraanizational  Effectiveness 

Organizational  effectiveness  is  one  of  the  most 
important  aspects  of  any  organization's  performance.  Steers 
refers  to  the  pursuit  of  organizational  effectiveness  as  the 
basic  responsibility  of  management  (Steers,  1976:55). 
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Furthermore,  an  org6mization's  level  of  effectiveness  is  a 

direct  measure  of  how  well  the  organization  is  performing 

its  mission,  and  it  is  usually  the  responsibility  of  the 

organizational  manager  to  ensure  the  organization  is 

performing  as  effectively  as  possible.  Albanese  contends: 

the  main  reason  for  m2magerial  positions  is  that  they 
presumably  contribute  to  improving  the  job  results  of 
other  people.  Through  their  own  job  performance, 
managers  are  supposed  to  help  others  perform  more 
efficiently  and  effectively.  .  .  Managers  are 
accountable  for  performance.  (Albanese,  1975:17) 

The  responsibility  of  the  contracting  squadron 

commander  is  no  exception.  Working  for  both  the  Logistics 

and  Wing  Commanders,  the  contracting  squadron  commander  is 

required  to  ensure  organizational  effectiveness.  It  is  for 

this  reason  that  the  organizational  manager,  and 

particularly  the  contracting  squadron  commander,  possess  a 

working  knowledge  of  the  concept  of  organizational 

effectiveness . 

Numerous  definitions  have  been  developed  to  describe 
organizational  effectiveness.  Anthony  and  Herzlinger 
believe  that  an  organization's  effectiveness  can  be  measured 
by  the  extent  to  which  its  outputs  accomplish  its  goals 
(Anthony,  1980:5).  Duncan  states  that  organizational 
effectiveness  is  more  than  efficiency.  The  effective  firm 
is  efficient;  it  is  adaptive  to  change;  and  it  maintains  a 
satisfactory  level  of  social  equilibrium  (Duncan,  1981:370). 
In  the  purchasing  context,  van  Weele  defines  organizational 


13 


effectiveness  as  the  relationship  between  actual  and  planned 
performance  (van  Weele,  1984:20).  For  a  contracting 
squadron,  organizational  effectiveness  may  be  viewed  from 
van  Weele *8  perspective.  For  exeunple,  a  contracting 
squadron  commander  may  set  a  goal  of  processing  500  purchase 
requests  over  the  ne:Kt  week.  In  order  to  achieve 
organizational  effectiveness  with  respect  to  this  goal,  the 
contracting  squadron  must  match  or  exceed  this  planned 
performance. 

The  importance  of  achieving  organizational 
effectiveness  cannot  be  denied.  Albanese  refers  to  the 
attainment  of  organizational  effectiveness  as  a  performance 
goal,  and  these  goals  represent  the  main  rationale  for  the 
existence  of  jobs  (Albanese,  1975:51).  To  achieve  this 
goal,  the  organization  must  be  evaluated  and  appraised 
properly  so  that  the  resulting  measures  provide  adequate 
feedback  to  the  organizational  manager.  Albanese  states 
that  in  order  to  be  useful  to  a  manager,  goals  should  be 
measurable,  otherwise  it  is  difficult  to  know  when  the  goal 
has  been  accomplished  (Albanese,  1975:54). 

To  help  determine  the  accomplishment  of  goals, 
mrnagement  should  rely  on  the  proper  metric  or  perfonaance 
evaluation  system  in  which  to  gauge  progress.  The  selected 
metric  or  performance  evaluation  system  must  use  the  proper 
performance  factors  to  provide  adequate  feedback  for  the 
manager  of  an  organization.  Only  through  sufficient 
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performance  evaluation  systems  will  the  manager  be  able  to 
achieve  the  bottom  line — organizational  effectiveness 
(Szilagyi  1980:457). 

Bgngfits  .Eai^grnflacfi 

According  to  Arjan  van  VIeele,  there  are  four  benefits 
to  evaluating  purchasing  performance.  First,  evaluation 
leads  to  better  decision  making  because  it  identifies 
variances  from  planned  results.  These  variances  can  be 
analyzed  to  determine  their  causes  and  action  to  take  to 
prevent  them  in  the  future.  Second,  evaluating  purchasing 
performance  makes  things  visible.  Regular  reporting  of 
actual  versus  planned  results  enables  the  individual  to 
verify  whether  his  or  her  expectations  have  been  realized 
and  provides  constructive  feedback  regarding  individual  and 
group  effectiveness.  Third,  measuring  purchasing 
performance  may  contribute  to  better  motivation.  Properly 
designed,  an  evaluation  system  can  meet  personal  and 
motivational  needs  of  the  employee  and  can  be  effectively 
used  in  a  constructive  goal  setting,  motivational,  or 
personal  development  program.  Fourth,  measuring  purchasing 
performance  may  lead  to  better  communication  between 
departments  as  well  as  other  organizations  and  in  the  long 
run,  improve  mutual  understanding  between  departments  and 
foster  a  team  effort  (van  WeeLe,  1984:18). 
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These  four  benefits  discussed  by  van  Weele  can  also 
apply  to  a  contracting  squadron,  once  a  contracting 
squadron  coitunander  identifies  substandard  performance,  a 
determination  can  be  made  as  to  what  caused  the  substandard 
performance  and  what  action  will  correct  it.  Reporting  the 
planned  versus  actual  performance  provides  a  squadron 
commander  the  opportunity  to  address  a  particular  area  and 
provide  feedback  with  respect  to  whether  the  actual 
performance  is  considered  effective.  Feedback  provided  to  a 
squadron  commander  through  performance  evaluation  can 
provide  a  means  to  increase  or  improve  motivation  within  the 
organization.  Finally,  if  a  contracting  squadron  commander 
is  aware  of  the  squadron's  performance  with  respect  to 
overall  effectiveness,  it  might  be  easier  and  more 
meaningful  to  discuss  perfomance  between  flights  and 
organizations. 

Performance  Evaluation  Systems 

In  reviewing  the  literature,  it  is  apparent  that  there 

are  many  different  performance  evaluation  systems  currently 

1 

in  place.  Numerous  benefits  and  limitations  are  associated 
with  these  diverse  systems  and  consequently  the  selected 
performance  evaluation  system  must  complement  the 
organization  it  is  evai”ating. 

Performance  evaluation  seeks  to  evaluate  the  overall 
success  of  the  chosen  course  of  action  and  to  identify  where 
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improvements  might  be  made  to  more  fully  realize  the 
projected  program  benefits  (Herbert,  1989:599).  In  an  Air 
Force  context,  if  a  contracting  squadron  has  200  open 
purchase  requests  in  its  possession  and  takes  action  to 
close  out  150,  the  resulting  performance  can  be  reviewed  to 
determine  what  course  of  acttor*  migiit  have  allowed  the 


squadron  to  close  out  all  200  purchase  orders. 


Many 


different  approaches  exist  for  assessing  organizational 
effectiveness.  Cameron  states  that  evaluators  have  used 
four  approaches  to  define  and  assess  organizational 
effectiveness.  These  approaches  are  (1)  goal,  (2)  system 
resource,  (3)  process,  and  (4)  strategic  constituencies 
(Cameron,  1980:68). 

The  goal  approach  is  the  most  widely  used  and  defines 
effectiveness  in  terms  of  how  well  an  organization 
accomplishes  its  goals.  Evaluators  using  this  approach 
focus  on  the  outputs  of  an  organization.  The  closer  the 
organization's  outputs  come  to  meeting  its  goals,  the  more 
effective  the  organization. 

Under  the  system  resource  approach,  an  organization's 
effectiveness  is  judged  on  the  extent  to  which  it  acquires 
needed  resources.  The  more  of  the  needed  resources  an 
organization  can  obtain  from  its  external  environment,  the 
more  effective  the  organization. 
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The  process  approach  holds  that  effective  organizations 
are  those  with  pn  absence  of  internal  strain,  whose  members 
are  highly  integrated  into  the  system,  whose  internal 
functioning  is  smooth  and  typified  by  trust  and  benevolence 
toward  individuals,  and  where  information  flows  smoothly 
both  vertically  and  horizontally.  Organizations  are  more 
effective  if  they  possess  a  greater  degree  of  these  internal 
characteristics,  less  effective  if  they  possess  a  lesser 
degree  of  these  characteristics. 

The  final  approach,  strategic  constituencies,  also 
referred  to  as  the  participant  satisfaction  model,  defines 
effectiveness  as  the  extent  to  which  all  of  the 
organization's  strategic  constituencies  are  at  least 
minimally  satisfied,  A  strategic  constituency  is  any  group 
of  individuals  who  have  some  stake  in  the  organization.  For 
example,  a  "itrategic  constituency  would  include  resource 
providers,  users  of  the  organization's  products  or  services, 
or  those  whose  lives  are  significantly  affected  by  the 
organization.  In  this  approach,  the  effectiveness  of  an 
organization  is  based  on  how  well  it  responds  to  the  demands 
and  expectations  of  its  strategic  constituencies.  Advocates 
of  the  strategic  constituencies  approach  have  usually 
emphasized  external  constituencies,  those  powerful  groups 
outside  the  organization  that  have  a  significant  impact  on 
its  functioning. 
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Each  of  the  four  approaches  identified  by  Cameron 
provide  useful  guidelines  for  systematically  assessing  the 
effectiveness  of  organizations ,  and  all  of  the  approaches 
could  be  considered  by  a  contracting  squadron  commander. 

The  goal  approach  is  especially  useful  when  organizational 
goals  are  clear,  consensual,  and  measurable.  The  goals  a 
contracting  squadron  commander  and  other  squadron  personnel 
set  and  their  attainment  are  the  keys  to  this  approaca. 

The  system  resource  approach  is  most  useful  when  there 
is  a  clear  connection  between  resources  received  by  the 
organization  and  what  it  produces.  For  a  contracting 
squadron  commander,  an  example  of  a  systems  resource  would 
be  manpower.  The  higher  the  number  of  personnel  assigned  to 
the  contracting  squadron,  the  higher  the  number  of  completed 
purchase  requests. 

The  third  approach,  process,  is  most  appropriate  when 
the  internal  processes  and  procedures  of  an  organization  are 
closely  associated  with  what  the  organization  produces,  or 
with  its  primary  task.  An  example  of  this  approach  in 
contracting  would  involve  a  contracting  squadron's  level  of 
effectivenaas  and  its  relationship  to  the  internal  processes 
accomplished  by  the  contracting  personnel  while  performing 
their  jobs. 

The  final  approach,  strategic  constituencies,  is  most 
appropriate  when  external  constituencies  have  a  powerful 
influence  on  the  organization's  operations  or  when  an 
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organization's  behavior  is  largely  reactive  to  strategic 
constituency  demands.  This  approach  is  probably  the  most 
applicable  to  contracting  squadrons  because  contracting 
squadrons  are  service  organizations  tasked  with  providing 
the  best  customer  support  possible. 

All  of  the  approaches  discussed  by  Cameron  are 
analytically  independent  and  therefore  only  one  of  them  is 
generally  appropriate  for  assessing  most  types  of 
organizations.  Identifying  which  approach  is  appropriate 
for  a  particular  organization  is  based  in  part  upon  the 
manner  in  which  the  organization  desires  to  measure 
effectiveness  (Cameron,  1980:79).  Regardless  of  the 
approach  adopted,  it  should  satisfy  the  user's  requirements 
and  be  accepted  as  valid  in  previous  efforts  (Campbell, 
1981:72). 

In  addition  to  the  model  suggested  by  Cameron,  recent 
literature  proposes  other  models  capable  of  evaluating 
purchasing  performance.  Pooler  suggests  that  one  model 
available  and  worthy  of  consideration  i:  Total  Recognition 
of  Environmental  and  Numerical  Development.  This  model 
states  that  the  best  standard  for  a  purchasing  department  is 
its  own  past  performance.  Data  regarding  such  performance 
factors  as  number  of  purchase  orders,  dollar  purchases  per 
year,  and  dollars  saved  per  jear  should  be  tracked  over  the 
last  five  years.  Using  this  data,  trend  analysis  can  be 
accomplished  and  the  findings  used  to  determine  the  overall 
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effectiveness  of  a  purchasing  department  (Pooler,  1973:69). 
Trend  analysis  is  one  method  that  is  presently  being  used  by 
many  operational  contracting  squadron  commanders.  The  BCAS 
report  provides  data  in  the  form  of  a  monthly  report  for 
many  different  performance  factors  present  within  the 
contracting  squadron.  Using  these  reports,  squadron 
commanders  can  analyze  and  evaluate  data  with  respect  to 
past  performance. 

Similar  to  the  model  presented  by  Pooler,  Croell 
contends  that  quantitative  measures  of  purchasing 
performance  should  be  used  as  the  basis  for  evaluating 
overall  purchasing  performance.  These  measures  should  be 
established  after  reviewing  the  particular  function  of  the 
purchasing  department  to  determine  which  areas  of 
performance  are  important  to  the  successful  completion  of 
the  department's  overall  mission.  Measures  identified  by 
Croell  as  important  include  dollar  purchases,  purchases  as  a 
percent  of  sales,  and  number  of  purchase  orders.  The 
performance  level  of  these  areas  could  then  be  used  to 
measure  the  effectiveness  of  the  purchasing  department 
(Croell,  1980:23). 

Van  Weele  refers  to  purchasing  effectiveness  as  the 
relationship  between  actual  and  planned  performance  and 
purchasing  efficiency  as  the  relationship  between  planned 
and  actual  sacrifices  made  in  order  to  realize  a  goal 
previously  agreed  upon.  Based  upon  these  definitions,  van 
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Weele  suggests  that  an  effective  performance  evaluation 
system  should  measure  both  purchasing  efficiency  and 
effectiveness.  Van  Weele  further  states  that  when 
establishing  a  performance  evaluation  system,  the  following 
guidelines  should  b  i  utilized.  First,  evaluation  systems 
should  be  designed  in  a  manner  that  corresponds  with  the 
daily  operations  of  the  buyer  involved.  Second,  the  buyer 
should  participate  in  the  ested>lishment  of  standards  for  his 
or  her  activities.  Third,  performance  feedback  should  be 
provided  to  the  buyer  on  a  timely  basis  so  this  information 
can  be  used  when  taking  corrective  action  when  necessary. 
Finally,  standards  for  evaluating  purposes  should  be  set 
only  in  those  areas  for  which  the  buyer  can  be  legitimately 
held  responsible  (van  Weele,  1984:18). 

Another  purchasing  evaluation  system  is  one  proposed  by 
Cavinato  who  contends  that  purchasing  effectiveness  should 
be  measured  from  the  viewpoint  of  non-purchasing  personnel 
who  are  considered  the  customer.  These  individuals  might  be 
from  the  engineering,  production,  accounting  or  other  non¬ 
purchasing  departments.  Cavinato  states  that  the  level  of 
effectiveness  within  the  purchasing  department  is  a  direct 
measure  of  the  level  of  customer  service  provided  in  the 
following  areas:  (1)  speed  of  obtaining  products  or  service; 
(2)  reliability  in  obtaining  products  or  services;  (3) 
perceived  effort  in  price/quality  options  search  and 
selection;  (4)  convenience  in  processing  requisitions;  (5) 
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advance  information  about  potential  supplier  deviations  from 
quality,  quantity,  or  delivery  commitments;  and  (6) 
assistance  with  special  problems  by  the  purchasing 
department  (Cavinato,  1987:10). 

Szilagyi  and  Wallace  state'  that  an  effective 
performance  evaluation  system  is  based  upon  reliable  and 
valid  performance  measures.  Reliability  is  a  product  of 
consistency  and  stability.  Consistency  demands  tliat  two 
alternative  methods  of  gathering  the  same  data  should  agree 
substantially  in  their  results.  Stability  demands  that  the 
same  measuring  device  should  provide  the  sane  results  time 
after  time,  as  long  as  the  characteristic  it  is  supposed  to 
be  assessing  doeu  not  change  (Szilagyi,  1980:449). 

Collins  and  Harris  point  out  that  a  shift  is  taking 
place  in  the  way  purchasing  performance  is  measured. 
Performance  evaluation  systems  are  moving  away  from  using 
traditional  purchasing  measures  and  in  their  place  are  using 
measures  that  encourage  continuous  improvement.  Traditional 
measures,  such  as  purchase  price  variance  were  once  the 
predominant  source  of  performance  evaluation  information. 
Purchasers  were  rewarded  when  their  price  was  lower  than  the 
standard  and  penalized  when  the  costs  were  above  the 
standard.  Using  rewards  and  punishment  in  this  manner 
caused  purchasers  to  spend  time  pursuing  methods  to  beat 
those  standards  rather  than  carrying  out  activities  they 
knew  were  important  (Collins,  1992:10). 
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In  a  sense,  the  same  principles  apply  in  the  Air  Force 
environment  where  many  operational  squadrons  commit  the 
majority  of  their  time  preparing  for  the  upcoming  Unit 
Effectiveness  Inspection  instead  of  concentrating  on 
providing  the  best  customer  support  possible.  Srikanth 
states: 

all  employees,  whether  they  be  managers,  supervisors, 
or  hourly  workers,  are  influenced  by  the  performance 
evaluation  measures  currently  used.  .  .  their  behavior 
and  performance  simply  reflect  the  standards  of  the 
existing  evaluation  system.  (Umble,  1990:12) 

Instead  of  traditional  measures,  m<  uy  managers  are 

selecting  meaningful  measures  that  are  more  representative 

of  the  work  performance  and  the  value  added  by  the 

purchasing  function  (Collins  1992:10). 

Srikanth  echoes  some  of  Collins'  and  Harris'  opinions, 

particularly  with  regard  to  traditional  measures.  Srikanth 

states  that  traditional  performance  measures  have  become 

obsolete  because  the  global  seller's  market  they  were 

designed  to  serve  no  longer  exists.  The  shortcoming  of 

these  measures  in  today's  buyer's  market  is  their  lack  of 

focus  on  competitive  elements.  Srikanth  also  states  that 

any  naw,  effective  system  of  measures  must  address  four 

critical  areas  that  are  lacking  in  traditional  measures. 

First,  the  system  should  be  able  to  track  customer 

satisfaction.  In  competitive  environment,  customers  must 

be  satisfied  or  market  share  will  decrease.  Second,  the 

system  should  be  able  to  track  relevant  financial 
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parfonnance.  The  new  system  must  be  able  to  answer  the 
basic  question:  Are  we  making  money?  Third/  the  system 
should  be  able  to  track  competitive  performance.  In  a 
competitive  environment,  customers  are  comparing  your 
performance' to  that  of  your  competition,  therefore,  every 
attempt  should  be  made  to  identify  competitors  and 
periodically  benchmark  their  performance  against  yours. 
Fourth,  the  system  should  be  ed^le  to  track  local  indicators 
to  see  how  effectively  sub-units  are  managing  materials  and 
resource  assets.  Examples  of  such  indicators  are 
inventories,  levels  of  scrap  and  rework,  and  overtime 
(Srikanth,  1992:51). 

In  addition  to  these  models,  the  Center  For  Advanced 
Purchasing  Studies  has  developed  a  model  based  upon 
benchmarking  techniques  which  provides  a  standard  or  point 
of  reference  to  measure  or  judge  quality,  value,  or 
competition.  Benchmarking  ultimately  provides  the  manager 
with  information  with  respect  to  best-in-industry 
performance  and  allows  for  improvement  to  be  made  in  less 
than  industry-best  areas  (feaman,  1992:8). 

gringiBi.9S..9i  an  Ett9gtim..Pgrt'<?nnflngs  Evajuatign 

System.  Only  certain  principles  found  within  these  models 
are  applicable  to  the  operational  contracting  squadron  when 
developing  performance  evaluation  systems.  Reasons  for  this 
include  the  following.  First,  contracting  squadrons  are 
non-profit  organizations  and  are  not  concerned  wi  i.u 


25 


achieving  a  profit  such  as  a  purchasing  department  withi 
the  civilian  sector.  Second,  contracting  squadrons  operate 
in  a  military  environment  that  is  regulated  by  stringent 
laws,  rules,  emd  regulations  that  must  be  followed. 

Finally,  due  to  force  reductions  and  reorganization, 
contracting  squadrons  are  often  forced  to  operate  with 
limited  personnel.  Considering  the  unique  environment  in 
which  a  contracting  squadron  functions,  the  following 
principles  should  be  considered  during  the  development  of  a 
performance  evaluation  system. 

1)  Performance  measures  should  embrace  continuous 
improvement . 

2)  Performance  measures  should  be  reliable  and 
valid. 

3)  Evaluation  systems  should  be  designed  with  the 
participation  of  the  user  and  should  be  relrted 
to  the  user's  daily  operations. 

4)  Effective  performance  evaluation  systems  should 
provide  a  method  to  assess  performance  of 
competitive  or  peer  organizations. 

5)  Measures  should  be  quantitative. 

The  remainder  of  the  chapter  will  examine  the  current 
inspection  system  employed  by  the  Air  Force,  as  well  as  the 
perceived  dissimilarities  between  the  Air  Force  system  and 
this  list  of  principles.  Finally,  the  chapter  will  address 
benchmarking  as  a  proposed  measure  that  may  be  useful  in 
eliminating  some  of  these  dissimilarities. 


Vi 
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Current  Air  Force  Evaluation  System 


The  performance  evaluation  system  currently  employed  by 
the  Air  Force  is  based  upon  principles  and  guidelines  found 
within  AFR  123-1,  The  Air  Force  Inspection  System,  The 
objective  of  the  Air  Force  Inspection  System  is  to  provide 
feedback  on  the  capability  of  Air  Force  units  to  perform 
their  assigned  missions.  More  specifically,  the  inspection 
system: 

(1)  measures  the  effectiveness  and  efficiency  of  units, 
functions,  programs,  and  guidance; 

(2)  evaluates  Air  Force  and  unit  internal  controls  and 
identifies  good  management  methods  as  well  as 
validating  and  comparing  management  infomnation  for 
possible  improvement; 

(3)  suggests  changes  in  resource  allocation  to  improve 
current  operations  or  to  plan  future  activities. 

(AFR  123-1,  1989:5) 

Air  Force  Regulation  123-1  uses  the  follov^ing  criteria 
to  assign  inspection  ratings  to  Air  Force  organizations. 
These  inspection  ratings  are  used  as  a  measure  of  overall 
organization  effectiveness.  The  ratings  are  as  follows: 

(1)  Outstanding.  Performance  or  operation  far  exceeds 
mission  requirements.  Procedures  and  activities  are 
carried  out  in  a  far  superior  manner.  Resources  and 
programs  are  very  efficiently  managed  zmd  are  of 
exceptional  merit.  Few  if  any  deficiencies  exist. 

(2)  Excellent.  Performance  or  operation  exceeds 
mission  requirements.  Procedures  and  activities  are 
carried  out  in  a  superior  manner.  Resources  and 
programs  are  very  efficiently  managed  and  relatively 
free  of  deficiencies. 

(3)  Satisfactory.  Performance  or  operation  meets 
mission  requirements.  Procedures  and  activities  are 
carried  out  in  an  effective  and  competent  manner. 
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Resources  and  programs  are  efficiently  managed.  Minor 
deficiencies  may  exist  but  do  not  impede  or  limit 
mission  accomplishment. 

<4)  Marginal.  Performance  or  operation  does  not  meet 
some  mission  requirements.  Procedures  and  activities 
are  not  carried  out  in  an  efficient  manner. 

Deficiencies  exist  that  impede  or  limit  mission 
accomplishment . 

(5)  Unsatisfactory.  Performance  or  operation  does  not 
meet  mission  requirements.  Procedures  and  activities 
are  not  carried  out  in  an  adequate  manner.  Resources 
and  programs  are  not  adequately  managed.  Significant 
deficiencies  exist  that  preclude  or  seriously  limit 
mission  accomplishment.  (AFR  123-1,  1989:6-7) 

Perceived  Limitations .  The  list  of  principles  used  to 

develop  the  performance  evaluation  system  currently  employed 

by  the  Air  Force  to  assess  operational  contracting  squadron 

performance  varies  dramatically  from  the  list  identified  by 

numerous  researchers  within  the  field  of  purchasing.  The 

Air  Force  holds  the  following  principles  important.  First, 

performance  criteria  are  most  effective  when  they  are 

developed  by  staff  functional  areas  and  subsequently 

employed  by  inspector  generals.  Second,  performance 

criteria  should  define  the  standards  against  which  the 

inspected  unit  and  functional  areas  are  compared  when 

determining  inspection  ratings  and  should  provide  inspection 

guidance  broad  enough  to  facilitate  inspector  judgment. 

Finally,  performance  criteria  should  define  a  logical 

balance  of  subjective  and  objective  assessment  (AFR  123-1, 

1989:7) . 

Another  aspect  of  the  Air  Force  evaluation  system 
involves  the  reliance  on  past  performance.  This  reliance 
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forces  an  organization  to  compare  itself  to  past  performance 
using  standards  established  by  personnel  other  than  the 
person  whose  performance  is  to  be  evaluated.  Because  of 
this,  the  evaluation  system  is  primarily  concerned  with 
performance  with  respect  to  established  standards,  rather 
than  concern  for  how  the  organization  is  performing  with 
regard  to  competitive  or  peer  organizations.  As  a  result, 
individuals  receiving  inspections  have  little  input  into  the 
system  that  is  ultimately  used  to  evaluate  their 
performance . 

Several  dissimilarities  exist  between  the  list  of 
principles  of  an  effective  performance  evaluation  system, 
and  the  principles  upon  which  the  Air  Force  evaluation 
system  is  based.  First,  the  Air  Force  system  is  not 
established  by  those  individuals  who  are  evaluated.  Second, 
the  Air  Force  system  does  not  provide  the  opportunity  for  an 
organization  to  compare  its  performance  against  its 
competitors  or  peers.  Finally,  the  current  Air  Force 
evaluation  system  does  not  embrace  continuous  improvement. 

A  performance  evaluation  system  based  upon  benchmarking 
principles  may  provide  the  meaus  to  eliminats  these 
dissimilarities  and  ultimately  improve  the  Air  Force 
inspection  system. 
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For  two  decades  after  World  War  II,  U.S.  industry  was 
not  challenged  by  foreign  'ompetition  in  any  market  and 
consequently  management  became  more  concerned  with  simply 
meeting  the  high  level  of  dememd  that  existed  in  most 
markets  rather  than  worry  about  integrating  quality 
managerial  and  business  techniques  into  their  operations 
(Umble,  1990:7).  In  1979,  one  such  corporation,  Xerox,  was 
astonished  when  a  Japanese  competitor  introduced  a  mid-size 
copier  for  under  $10,000,  an  amount  less  than  it  cost  Xerox 
to  produce  a  similar  copier.  Convinced  that  the  competitor 
had  priced  the  copier  below  fair  market  value,  Xerox  began 
an  earnest  benchmarking  study  to  determine  the  meaning 
behind  the  market  price.  The  results  of  the  study  were 
staggering  to  the  leaders  of  the  Xerox  Corporation.  The 
study  indicated  that  the  Japanese  competitor  was  radically 
more  efficient  in  production  operations  (Port,  1992:74). 
Because  of  Xerox'  success  with  identifying  theii* 
competitor's  new  processes,  manufacturing  components,  and 
costs  of  manufacturing,  senior  management  decided  to 
incorporate  benchmarking  as  a  corporate-wide  effort. 

"Today,  Xerox  has  regained  market  share,  dramatically 
lowered  cost  emd  improved  quality,  and  saved  itself  from 
financial  disaster"  (Pryor,  1989:28). 

Prior  to  this  revelation  by  the  Xerox  Corporation,  most 
manufacturing  and  business  firms  employed  traditional 


measures  consisting  Of  internally  developed  standards 
utilizing  budgeting  procedures  with  adjustments  for 
productivity  and  customer  satisfaction  (Camp,  1989:7).  For 
the  first  time  the  benefits  associated  with  benchmarking 
were  realized  and  benchmarking  has  become  a  common  component 
of  the  many  Total  Quality  Management  programs  adopted  by 
large  companj.es  since  the  introduction  of  benchmarking 
(Rothman,  1992:64). 

Definition.  The  formal  definition  of  benciimarking  is 
"the  continuous  process  of  measuring  products,  services,  and 
practices  against  the  toughest  competitors  or  those 
companies  recognized  as  industry  leaders"  (Camp,  1989:10). 
Simply  stated,  benchmarking  is  the  comparison  of  a  given 
business  function  across  companies  and  can  be  defined  as  ( 1 ) 
measuring  your  perfoxmance  against  that  of  be8t~in~class 
companies;  (2)  determining  how  the  best-in-class  achieve 
those  performance  levels;  and  (3)  using  the  information  as 
the  basis  for  your  own  company's  targets,  strategies,  and 
implementation  (Pryor,  1989:28). 

Types.  Many  different  types  of  benchmarking  have  been 
identified  within  current  literature.  Pryor  has  identified 
three — strategic,  operational,  and  business  management 
benchmarking.  Strategic  benchmarking  involves  the 
comparison  of  different  market  strategies  and  a  correlation 
of  those  strategies  to  marketplace  success.  Operational 
benchmarking  focuses  on  a  specific  aspect  of  a  company's 
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functional  operations  and  identifies  ways  to  achieve  best- 
in-class  status.  Finally,  business  management  benchmarking 
involves  traditional  support  functions  that  are  benchmarked 
in  order  to  evaluate  the  worth  of  that  function  to  the 
overall  corporation  (Pryor,  1989:28). 

In  addition  to  these  three  types,  Camp  identified  three 
additional  forms  of  benchmarking:  (1)  benchmarking  against 
internal  operations;  (2)  competitive  benchmarking;  and  (3) 
generic  benchmarking.  Benchmarking  against  internal 
operations  relates  to  the  comparison  of  similar  functions  of 
internal  operations  within  the  same  organization. 

Competitive  benchmarking  involves  the  comparison  of  direct 
product  competitors.  Its  objective  is  to  show  what  the 
competitive  advantages  zmd  disadvantages  are  between  direct 
competitors.  Generic  benchmarking  refers  to  the  comparison 
of  business  functions  or  processes  that  are  the  same 
regardless  of  industry  (Camp,  1989:254). 

Benefits.  Traditionally,  organizations  have 
concentrated  on  tracking  and  analyzing  internal  trends  as  a 
means  of  evaluating  performance.  By  doing  so,  the 
organization  is  actually  comparing  itself  to  its  past 
performance.  While  this  is  a  valid  form  of  performance 
measurement,  it  does  not  provide  information  as  to  how 
performance  compares  to  that  of  its  peers  (Seaman,  1992:8). 
Benctmarking  can  help  address  this  problem  by  allowing  an 
organization  to  compare  itself  to  the  best-in-class 
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organizations,  quantify  differences  in  performance,  document 
why  those  differences  exist,  and  identify  steps  to  catch  up 
to  and  surpass  the  best-in-class  (Pryor,  1989:29). 

Camp  discusses  five  benefits  to  benchmarking.  They 
are:  (1)  aid  the  organization  in  meeting  customer 
requirements;  (2)  aid  the  organization  in  establishing 
effective  goals  and  objectives;  (3)  provide  a  true  measure 
of  productivity;  (4)  enable  the  organization  to  become 
competitive;  and  (5)  aid  the  organization  in  incorporating 
industry-best  practices  (Camp  .1989: 27 ) .  Table  2.1 
summarizes  Camp's  rationale  for  benefits  of  benchmarking. 
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Table  2.1.  Benefits  Associated  with  Benchmarkim 


Many  of  the  benefits  listed  in  Table  2.1  incorporate 
the  concepts  and  perceived  criteria  needed  for  an  effective 
performance  evaluation  system.  For  instance,  as  discussed 
in  Chapter  I,  the  Air  Force  is  currently  pursuing  total 
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quality  initiatives  in  which  the  focus  is  for  Air  Force 
organizations  to  become  competitive  and  highly  committed  to 
new  ideas  of  proven  practices  and  technology.  Furthermore, 
the  Air  Force  is  continually  searching  for  industry-’best 
practices  with  proactive  ideas  resulting  in  superior 
performance.  Finally,  for  an  org^mization  to  effectively 
perform  its  mission  involves  achieving  customer  satisfaction 
and  this  can  only  be  accomplished  by  defining  the  customer's 
requirements.  Regardless  of  who  the  organization's 
customers  are,  the  leader  of  an  organization  must  achieve 
customer  satisfaction  through  the  use  of  quality 
initiatives.  Benclimarking  techniques  satisfy  all  of  these 
objectives  and  provide  the  manager  of  an  organization  the 
necessaiy  tools  needed  to  evaluate  organizational 
performance . 

Despite  the  fact  that  there  are  many  benefits 
associated  with  benchmarking,  the  results  of  a  survey  taken 
in  1991  of  87  member  companies  of  the  International 
Benchmarking  Clearinghouse  found  that  while  "79  percent  of 
those  polled  believed  companies  will  have  to  benchmark  to 
survive,  95  percent  believed  that  most  companies  don't  know 
how  to  do  it"  (Biesada,  1992:34).  The  results  of  this 
survey  indicate  that  most  managers  understand  the  importance 
of  benchmarking,  but  fail  to  incorporate  benchmarking  as  a 
means  to  solve  quality  problems.  Only  recently  has  the  Air 
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Force  been  involved  with  pursuing  its  own  quality  programs, 
such  as  benchmarking  (Danis,  1993:4). 


Process.  According  to  Camp,  the  benchmarking  process 
can  be  separated  into  five  distinct  phases.  These  phases 
are:  (1)  planning;  (2)  analysis;  (3)  integration;  (4) 
action;  and  (5)  maturity  (Camp,  1989:17).  These  phases  are 
summarized  in  Table  2.2. 


Table  2.2.  Camp's  Five  Phase  Process  for  Benchmarking 

I .  Planning 

a.  identify  what  is  to  be  benchmarked 

b.  identify  comparative  companies 

c.  determine  data  collection  method  and  collect 
data 

II.  Analysis 

a.  determine  current  performemce  "gap" 

b.  project  future  performance  levels 

III.  Integration 

a.  communicate  benchmark  findings  and  gain 
acceptance 

b.  establish  functional  goals 

IV .  Action 

a.  develop  action  plans 

b.  implement  specific  actions  and  monitor 
progress 

c.  recalibrate  benchmarks 

V.  Maturity 

a.  leadership  position  attained 
_ b.  practices  fully  integrated  into  processes 


The  first  phase,  planning,  consists  of  three  steps. 

The  initial  step  is  to  determine  what  is  to  be  benchmarked. 
To  make  this  determination,  the  product  of  the  business  must 
be  identified.  This  can  be  accomplished  by  analyzing  the 
organization's  mission  statement  to  determine  the  outputs 
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for  which  the  organization  is  responsible.  Ultimately,  the 
outputs  that  should  be  benchmarked  are  those  that  are 
measured^le,  have  data  readily  available,  and  will  provide 
useful  information  to  organizational  managers  in  their  quest 
to  improve  the  overall  effectiveness  of  the  organization. 
Once  a  determination  has  been  made  as  to  what  factors  are  to 
be  benchmarked,  the  next  step  is  to  identify  comparative 
companies  to  benchmark  against.  Leader  companies  or 
organizations  should  be  used  to  compare  performance  against 
because  by  doing  so,  superiority  will  be  ensured.  The  third 
step  in  the  planning  phase  is  to  determine  an  appropriate 
data  collection  method  and  to  collect  the  data.  There  are 
an  infinite  variety  of  ways  to  obtain  the  required  data,  but 
a  combination  of  methods  that  best  meets  the  study  needs 
will  most  often  be  productive. 

Once  the  required  data  has  been  collected,  the 
benchmarking  process  can  move  into  the  analysis  phase.  The 
first  step  to  data  analysis  is  to  determine  the  current 
performance  gap.  This  can  be  accomplished  by  actually 
assessing  one's  performance  against  other  organizations  to 
determine  if  the  competitor  is  a  better  performer.  If  the 
determination  is  made  that  the  competitor  is  a  better 
performer  then  the  questions,  "By  how  much?"  and  "Why?" 
should  oe  asked.  The  performance  gap  may  be  positive  (no 
other  organization  is  superior),  negative  (one  or  more 
organizations  are  superior),  or  one  of  parity  (all 
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organizationB  are  performing  at  the  same  level). 

Determining  the  performance  gap  enables  the  organization  to 
move  into  the  second  step  of  analysis — projecting  future 
performance  levels.  In  this  step,  the  organization  must  now 
decide  whether  its  future  performance  should  be  directed 
toward  closing  the  gap  or  to  capitalize  on  a  positive  gap. 

The  third  phase  of  benchmarking,  integration,  consists 
of  two  steps.  The  first  step  is  to  communicate  benchmark 
findings  and  gain  acceptance  throughout  the  entire 
organization.  To  accomplish  this,  findings  must  be  clearly 
and  convincingly  demonstrated  as  being  correct  and  based 
upon  substantive  data.  Once  findings  have  been  accepted, 
they  can  be  converted  into  a  statement  of  operational 
principles  or  functional  goals. 

The  fourth  phase  of  the  benchmarking  process  is  action. 
The  action  phase  uses  those  principles  and  goals  identified 
in  the  integration  phase  and  attempts  to  convert  them  to  an 
action  process.  Additionally,  periodic  measurement  and 
assessment  of  achievement  must  be  put  into  place  as  well  as 
provisions  for  updating  and  recalibrating  the  established 
benchmarks . 

The  last  phase  of  benchmarking  is  that  of  maturity 
which  is  reached  when  industry-best  practices  are 
incorporated  in  all  organizational  processes  thereby 
enabling  the  organization  to  reach  a  position  of  leadership. 
In  this  phase,  continuous  improvement  is  essential  as  the 
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goal  of  benchmarking  ultimately  entails  incorporation  of 
industry-best  practices  and  achieving  organizational 
superiority  (Camp,  1989:17). 

figncJlMBiong 

As  identified  earlier  in  this  chapter,  dissimilarities 
exist  eunong  the  key  principles  found  within  the  Air  Force 
evaluation  system  and  those  used  by  purchasing  departments 
within  Industry.  First,  performance  criteria  used  by  the 
performance  evaluation  system  should  be  developed  and 
established  by  the  buyer  or  individual  whose  performance 
will  be  evaluated.  This  evaluation  should  not  be 
accomplished  at  a  level  several  tiers  above  the  user  level, 
such  as  the  current  Air  Force  inspection  system  which 
develops  the  system  at  the  functional  staff  level.  Camp 
states  that  because  every  function  of  a  business  should  be 
considered  a  candidate  for  benchmarking,  there  is  a  critical 
need  for  senior  management  to  orchestrate  the  benchmarking 
effort.  Furthermore,  to  ensure  effectiveness,  benchmarking 
should  ultimately  be  conducted  by  the  people  who  will  use  it 
(Camp,  19S9b:£6}.  To  solve  this  problem,  benchmarking 
provides  an  evaluation  system  developed  and  implemented  by 
those  users  who  would  be  directly  affected  by  the  system. 

Second,  effective  performance  evaluation  systems  should 
provide  the  opportunity  for  the  organization  to  compare 
performance  against  its  competitors  and  peers.  This  concept 
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involves  the  fundamental  precept  upon  which  benchmarking  is 
founded.  Benchmarking  is  "the  continuous  process  of 
measuring  products,  services,  and  practices  against  the 
toughest  competitors  or  those  companies  recognized  as 
industry  leaders"  (C.amp,  1989:10}.  Implementing 
benchmarking  as  an  alternative  performance  evaluation  system 
would  allow  organizations  within  the  Air  Force  to 
incorporate  Air  Force  best  practices  and  thus,  internalize 
total  quality  initiatives  as  a  means  to  improve  overall 
performance. 

Finally,  effective  performance  evaluation  systems 
should  embrace  continuous  improvemi  xit .  In  this  sense, 
benchmarking  involves  a  continuous  process  of  identifying 
the  factors  to  be  benchmarked,  collecting  the  required  data, 
establishing  the  benchmarks,  determining  the  performance 
gap,  communicating  the  benchmarking  findings,  developing 
action  plans,  implementing  specific  actions,  and 
periodically  assessing  achievement  of  the  benchmark  and 
recalibrating  or  updating  the  benchmark.  The  process 
continues  until  the  organization  has  reached  a  level  of 
industry  leadership,  and  even  then,  inuat  continue  to 
benchmark  in  order  to  remain  superior.  Similarly,  the  Aj>.r 
Force  must  also  address  the  issue  of  continuous  improvement, 
and  benchmarking  would  provide  the  means  of  furnishing 
feedback  to  the  organizational  leaders  as  a  basis  for 
improved  organizational  decisions. 
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III. 


Overview 

The  purpose  of  this  chapter  is  to  identify  and  discuss 
the  methodology,  procedures,  and  activities  used  in 
developing  an  alternative  perfoxrmance  evaluation  system. 
First,  the  chapter  reviews  the  investigative  questions  which 
form  the  basis  of  the  thesis  and  provide  a  focus  for  the 
research.  Second,  the  chapter  identifies  the  type  of 
research  design  used  to  guide  the  study.  Discussion  of 
design  involves  the  selection  of  performance  factors, 
population  and  sample  selection,  and  limitations  associated 
with  the  research.  In  addition,  research  design  will  also 
address  the  survey  instrument  .in  detail— its  development, 
structure,  and  overall  purpose.  Finally,  the  chapter  will 
conclude  with  an  in-depth  analysis  of  each  of  the 
investigative  questions. 

Restatement  ,.al  Invgg.ti.WiYe  Quggtigna 

Four  investigative  questions  provide  a  focal  point  for 
the  remainder  of  this  chapter. 

1.  What  performance  factors  currently  exist  within  a 
typical  contracting  squadron? 

2.  Which  of  these  performance  factors  are  considered 
significant  for  measuring  organizational  performance  and 
therefore  should  be  included  in  the  development  of  an 
effective  performance  evaluation  system? 

3.  Can  a  benchmark  be  established  for  each  significant 
performance  factor? 
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4.  Can  a  vicible  performance  evaluation  system,  capable  of 
accurately  measuring  overall  effectiveness  for  a  typical 
contracting  squadron,  be  developed  using  benchmarking 
techniques  and  concepts? 

RggsaEgh 

The  purpose  of  research  design  is  to  provide  the 
technical  aspects  of  the  study  and  provide  a  means  to 
document  the  methodology  of  the  study  so  that  findings  can 
be  replicated.  This  portion  of  the  chapter  will  cover  the 
following  areas:  selection  of  performance  factors, 
population  and  sample  selection,  survey  development,  survey 
instrument,  and  limitations  of  the  research  design. 

Selection  of  Performance  Factors.  A  review  of 
literature  was  conducted  to  identify  all  potential 
performance  factors  capable  of  measuring  contracting 
squadron  performance. 

The  first  step  needed  to  satisfy  this  objective 
involved  establishing  a  potential  list  of  all  performance 
factors  that  currently  exist  within  a  typical  contracting 
squadron.  Development  of  this  list  required  an  extensive 
literature  review  and  discussions  with  operational 
contracting  personnel. 

Available  literature  pertaining  to  contracting 
performance  measures  was  reviewed  to  create  an  initial 
listing  of  performance  factors  found  within  a  typical 
contracting  squadron.  Literature  used  in  this  search 
included  the  following:  the  IG  Checklist  dated  16  March 
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1991;  AFLMC  Report  LC870309,  Hanagement  Information  Needs 
for  the  Base  Contracting  Office,  March  1989;  monthly  BCAS 
data  reports  and  Management  Critiques  from  K.I.  Sawyer  AFB; 
Strategic  Air  Command  Statistics  Pamphlet  (Oct  91-Feb  92); 
and  the  BCAS  Users  Guide. 

In  addition  to  the  literature  review,  input  was 
solicited  from  contracting  personnel  within  the  Air  Force 
who  had  operational  experience  in  the  field  of  contracting. 
These  individuals  included  present  and  prior  contracting 
squadron  commanders  and  deputy  base  contracting  officers, 
contracting  staff  personnel  at  Air  Combat  Command,  and 
professors  of  Contract  Management  at  the  Air  Force  Institute 
of  Technology  (AFIT). 

The  results  of  the  literature  review  aund  personnel 
interviews  were  a  compilation  of  63  performance  factors  that 
are  currently  used  to  measure  performance  within  a 
contracting  squadron.  These  63  performance  factors  can  be 
found  highlighted  in  bold  print  in  Appendix  B — Survey 
Instrument.  This  list  of  factors  form  the  basis  of  the 
survey  questionnaire. 

Population  and  Sample  Selection.  As  discussed  in 
Chapter  II,  one  of  the  perceived  limitations  involving  the 
current  Air  Force  inspection  system  is  that  the  users  fail 
to  develop  the  system.  Therefore,  any  input  regarding  the 
performance  factors  to  be  included  in  an  evaluation  system 
should  be  provided  by  the  users.  This  perceived  limitation 
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was  addressed  when  selecting  the  representative  population 
and  sample  to  be  studied. 

The  selection  of  the  population  and  sample  were 
determined  with  regard  to  accurate  representation  of  the 
characteristics  of  the  population  to  be  represented.  The 
population  of  concern  for  this  research  effort  was 
determined  to  be  all  contracting  squadron  commanders,  deputy 
base  contracting  officers,  emd  executive  noncommissioned 
officers  assigned  to*  any  of  the  major  commands  located 
within  the  GONUS.  These  major  commands  include:  Air  Combat 
Comraeuad  (ACC);  Air  Mobility  Command  (AMC);  Air  Force 
Materiel  Command  (AFMC);  Air  Training  Command  (ATC),  Air 
University  (AU);  and  Air  Force  Space  Command  (AFSC). 

Emory  and  Cooper  (1991),  state  the  sampling  technique 
selected  should  depend  upon  the  requirements  of  the  project, 
its  objectives,  and  the  availability  of  funds.  Since  one  cf 
the  objectives  of  this  study  is  to  determine  the  general 
attitudes  of  contracting  personnel  with  respect  to 
performance  factors  that  are  important  to  effective 
contracting  squadrons,  quota  sampling  was  selected  to  assure 
that  the  sample  was  representative  of  the  population  from 
which  it  was  drawn  (Emozy,  1991:275).  Quota  sampling  is  a 
form  of  nonprobability  sampling  in  which  certain  criteria 
are  imposed.  For  this  study,  the  researchers  imposed  the 
criteria  specifying  that  the  selected  sample  should  provide 
meaningful  input  for  the  selected  topic,  and  that  the  sample 
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represents  a  distribu  -ion  in  the  population  that  is  easily^ 
estimated.  The  resulting  sample  size  totaled  258  people, 
which  is  considered  large  with  respect  to  statistical 
analysis  (McClave,  1991:85).  Appendix  C  contains  a  list  of 
all  the  operational  contracting  squadrons,  by  base,  that 
were  sampled. 

Upon  determining  the  population  and  sample,  the 
decision  was  then  made  to  utilize  a  survey  instrument  to 
gather  information  necessary  for  this  thesis  study.  The 
reason  for  selecting  a  survey  instrument  to  satisfy  the 
objectives  of  the  study  is  because  us'e  of  a  survey 
instrument  provides  the  most  feasible  method  to  answer  all 
of  the  investigative  questions,  and  for  this  research  the 
respondents  are  uniquely  qualified  to  provide  the  desired 
input  needed  tc  answer  the  investigative  questions.  Other 
methods  of  collecting  pertinent  data  were  considered  and  the 
decision  was  made  to  develop  an  original  survey  instrument 
that  specifically  fit  the  needs  of  the  research. 

Furthermore,  the  decision  was  also  made  to  utilize  mail 
surveys  to  satisfy  the  objectives  of  the  study.  This 
Hg^ision  WAS  on  sovssrsiX  fACwCaTs 

most  important  attributed  to  funding  and  time  constraints 
placed  upon  the  research  effort,  and  the  inaccessibility  of 
the  majority  of  the  selected  sample.  In  addition,  mail 
surveys  are  considered  to  be  extremely  practical  because 
they  allow  for  anonymity  for  the  respondent,  and  provide 
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more  time  for  the  respondent  to  consider  the  questions  and 
their  responses  (Emory,  1991:333). 

To  effectively  consider  all  impacts  associated  with 
mail  surveys,  the  authors  also  considered  various  methods 
for  improving  return  rates.  Receiving  a  high  return  rate  is 
important  because  the  higher  the  retuinn  rate,  the  more 
accurate  the  estimate  will  tend  to  be  (McClave,  1991:85). 

In  an  effort  to  improve  the  return  rate  the  following  areas 
were  addressed:  inclusion  of  self-addressed  return 
envelopes,  cover  letter  clearly  explaining  the  benefits  of 
the  study,  assurance  of  anonymity  for  the  respondent,  and 
deadline  due  date. 

Reasons  for  limiting  the  sample  to  only  those  bases 
located  within  the  CONUS  were  threefold.  First,  contracting 
procedures  and  regulations  vary  between  CONUS  and  overseas 
contracting  squadrons.  This  inherent  variation  may  create 
significant  differences  among  the  perceptions  of  those 
individuals  assigned  to  either  a  CONUS  or  overseas  location. 
Additionally,  this  variation  may  result  in  differences  in 
the  manner  in  which  the  different  contracting  squadrons 
determine  the  level  of  importance  among  different 
performance  factors.  Second,  due  to  limited  response  time, 
monetary  considerations,  and  the  long  mail  time  often 
associated  with  overseas  installations,  the  researchers 
determined  that  an  overseas  sample  may  not  be  feasible. 
Finally,  analysis  of  only  CONU.S  installations  would  still 
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provide  statistically  significant  results  due  to  the  large 
sample  size.  Large  sample  statistics  involve  smaller 
sampling  errors,  greater  reliability,  and  increases  the 
power  of  the  statistical  tests  applied  to  the  data  (Isaac, 
1985:189).  Furthermore,  the  Central  Limit  Theorem  assures 
us  that,  for  a  large  sample  test  of  hypothesis,  "the  test 
statistic  will  be  approximately  normally  distributed 
regardless  of  the  shape  of  the  underlying  probability 
distribution  of  the  population"  (McClave,  1991:356). 

Survey  Developmant.  In  order  to  effectively  evaluate 
the  selected  sample,  the  authors  developed  a  survey 
instrument  that  was  used  to  collect  data  to  answer  the 
research  questions.  Isaac  (1985),  states  that  the  guiding 
principles  underlying  surveys  is  that  they  should  be 
systematic,  representative,  objective,  and  quantifiable. 
These  four  principles  ware  critically  considered  during  the 
survey  development. 

Development  of  the  survey  involved  several  steps  to 
ensure  a  systematic,  representative  instrument  capable  of 
measuring  the  essential  areas  needed  for  the  research.  The 
first  step  involved  including  ail  of  the  63  performance 
factors  that  were  previously  identified  thro,^  ^h  the 
research.  Next,  it  was  discovered  that  another  Graduate 
Contract  Management  thesis  team  was  conducting  a  similar 
research  study.  This  team  was  interested  in  determining 
performance  factors  needed  for  measuring  productivity  and 
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efficiency.  Since  the  other  thesis  team  was  considering  the 
same  population  sample,  and  their  selected  performance 
factors  were  very  similar  to  ones  identified  within  this 
research,  the  determination  was  made  to  mail  out  a  joint 
survey  capable  of  collecting  information  required  for  both 
thesis  teams.  This  decision  was  based  upon  the  perceived 
threats  that  may  arise  with  two  separate  surveys  using 
similar  subjects  and  format,  and  attempting  to  gather 
information  from  a  similar  sample.  Furthermore,  mailing  out 
two  similar  surveys  to  the  seuae  sample  may  introduce  biased 
results  for  either  the  number  of  overall  survey  responses 
received,  or  actual  question  responses  themselves. 

Once  the  initial  survey  was  drafted,  a  pretest  was 
conducted  using  ten  individuals  within  the  AFIT  Graduate 
Contract  Management  section  who  had  operational  contracting 
squadron  experience  as  either  squadron  commanders  or  flight 
chiefs.  Fink  (1985),  states  that  the  purpose  of  a  pretest 
is  threefold:  (l)  measure  the  practicality  of  the  survey; 

(2)  determine  the  validity  of  the  survey;  and  (3)  interpret 
the  reliability  of  the  survey. 

Practicality  of  the  survey  alludes  to  a  determination 
as  to  the  interpretability  of  the  survey.  Upon  review  of 
the  survey  responses  from  the  pilot  group,  it  was  determined 
that  all  directions,  questions,  and  instructions  were  clear 
because  all  surveys  were  completed  as  anticipated  without 
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questions  or  reques^^s  for  clarification  from  the  pilot 
qroup . 

The  second  purpose  of  the  pretest,  validity,  refers  to 
whether  the  survey  measures  what  it  is  supposed  to  measure. 
Results  gathered  during  the  pretest  of  the  survey  supported 
the  fact  that  the  survey  did  gather  attitudinal  responses 
from  the  respondent  with  respect  to  determining  the 
performance  factors  that  are  importeuit  to  the  overall 
effectiveness  of  on  operational  contracting  squadron. 

The  final  purpose  of  the  pretest  was  to  assess  the 
reliability  of  the  survey-  Reliability  of  the  survey  refers 
to  the  accuracy  and  precision  of  the  measurement  procedure. 
Since  the  results  received  by  all  ten  members  of  the  pilot 
group  were  fairly  consistent  in  their  responses,  it  gave  the 
indication  that  the  survey  was  also  reliable. 

In  addition  to  the  review  conducted  by  the  pilot  group, 
the  survey  was  also  reviewed  for  validation  purposes  by  the 
Air  Force  Logistics  Management  Agency  at  Gunter  AFB.  One  of 
the  main  responsibilities  of  this  office  is  identifying, 
researching,  and  solving  problems  facing  operational 
contracting  squadrons.  This  review  did  not  result  in  any 
substantive  changes  to  the  list  of  performance  factors. 

As  a  result  of  the  initial  reviews  and  feedback 
received  from  the  pilot  test,  no  changes  were  made  to  the 
original  list  of  performance  factors.  The  final  si\rvey 
instrument  is  found  in  Appendix  B. 
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Survey  Instrument.  Part  I,  Background,  was  designad  to 
collect  demographic  information  which  could  be  used  to 
determine  the  individual  differences  among  the  respondents < 
In  this  part,  the  respondents  were  asked  to  provide 
information  regarding  their  experience  level  with  respect  to 
total  contracting  experience,  total  experience  in  their 
present  position,  major  command  to  which  they  are  assigned, 
and  opinions  concerning  performance  evaluation  systems. 

Part  II,  Organizational  Performance  Factors,  contains  a 
list  of  83  performance  factors  to  be  rated  by  the 
respondent.  Because  the  survey  was  a  compilation  of  two 
different  research  efforts,  some  of  these  performance 
factors  measured  effectiveness,  while  others  measured 
productivity  or  efficiency.  Only  those  performance  factors 
that  measure  effectiveness  were  of  coxicern  in  this  thesis 
effort. 

To  effectively  measure  performance,  a  five  point  Likert 
scale  was  developed  to  provide  input  for  analysis.  This 
scale  was  chosen  because  it  can  bo  treated  as  an  interval 
scale  which  provides  the  opportunity  to  use  paremetric 
analysis  techniques.  These  techniques  include  determining 
the  mean  response,  standard  deviation  for  each  question,  and 
tests  for  significance  between  populations  (McClave, 
1991:307).  The  following  scale  was  included  on  the  survey 
instrument:  (1)  definitely  nov  important;  (2)  somewhat  not 


importemt;  (3)  neutral;  (4)  t^oinewhat.  important;  and  (5) 
definitely  important. 

Part  III,  Additional  Performance  Factors,  provided  a 
means  for  the  respondent  to  add  any  additional  performance 
factors  they  determined  important  for  measuring  contracting 
performance  which  wore  not  included  among  the  original  list 
of  63.  Inclusion  of  this  pai-t  was  twofold.  First, 
receiving  input  from  the  respondents  validates  the  initial 
list  of  performance  factors  developed  by  the  researchers. 
Second,  write-in  responses  allow  the  researchers  to  expand 
the  list  of  performance  factors  to  include  any  additional 
factors  that  may  be  important  in  measuring  performance. 

Part  IV,  Overall  Performance,  asked  the  respondents  to 
select,  from  the  list  of  63  factors  included  on  the  surv'ey 
and  any  additional  factors  included  in  Part  III,  the  five 
performance  factors  they  felt  were  the  most  important  in 
measuring  organizational  performance  within  an  operational 
contracting  squadron.  To  provide  effective  feedback,  the 
respondents  were  asked  to  rank  order  these  five  factors  in 
order  from  first  in  importance  to  fifth  in  importance.  This 
rank  order  will  provide  necessary  information  used  for 
identifying  the  most  crucial  factors  in  measuring 
performance. 

Part  V,  Overall  Performance  Of  Your 
Squadron/Organization,  asked  the  respondent  to  rank  the 
overall  performance  of  their  assigned  organization.  The 
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answer  choices  were  detezinined  from  a  segmented  scale  that 
range  from,  Unsatisfactory  to  Outstanding.  The  respondents 
were  asked  to  answer  this  part  based  on  their  personal 
perception  concerning  the  effectiveness  of  their 
organization,  and  not  to  consider  the  input  received  from 
their  last  16  rating  or  command  e  valuation.  The  purpose  of 
this  part  was  to  provide  the  opportunity  to  evaluate  the 
individual's  preference  of  most  important  performance 
factors  to  how  well  they  considered  their  squadron  to  be 
performing. 

Part  VI,  Desirable  Characteristics  of  a  Performance 
Evaluation  and  Feedback  System,  was  placed  on  the  survey  by 
the  other  thesis  team.  The  information  provided  by  the 
respondent  in  this  part  was  of  no  concern  for  this 
particular  research  effoirt. 

Part  VII,  Additional  Comments,  provided  the  survey 
respondent  the  opportunity  to  make  any  additional  comments 
they  felt  were  ii  portant  to  the  research  effort  and  to 
request  a  copy  of  the  completed  thesis.  Although  this 
section  has  little  statistical  value  for  the  thesis  effort, 
the  comments  may  point-  out  strengths  and  weaknesses  of  the 
survey  instrument  or  the  research  effort  in  general. 

Limitations  of  Research  Pegian.  Several  potential 
limitations  exist  with  the  research  design  utilized  for  this 
study.  One  potential  source  of  error  lies  in  the  fact  that 
oftentimes  respondents  only  respond  to  mail  surveys  in  which 
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the  respondents  feel  comfortable  with  the  survey  questions 
^md  material  (Emory,  1991:333).  For  example,  if  an 
organization  recently  received  an  Unsatisfactory  or  Marginal 
rating,  then  there  may  be  a  higher  probability  that  these 
individuals  would  not  return  a  survey  dealing  with 
performance  factors  since  they  might  not  be  as  interested  in 
the  topic.  On  the  other  hand,  those  individuals  that  were 
recently  involved  with  receiving  a  Satisfactory  or  higher  IG 
rating  are  most  likely  to  be  interested  in  the  survey 
content  and  more  apt  to  participate  in  the  research  study. 
Based  on  these  predicted  inputs,  the  results  of  the  mail 
survey  may  be  skewed  and  biased  providing  a  potential 
limitation  for  the  research.  This  potential  limitation  will 
be  analyzed  in  Chapter  IV,  by  presenting  the  demographic 
information  for  all  the  survey  respondents  and  drawing 
conclusions  about  the  results  indicated. 

Another  potential  limitation  involves  the  constraints 
imposed  upon  performance  factor  selection.  The  list  of  63 
performance  factors  was  limited  to  those  quantitative 
factors  that  were  both  measurable  and  available  to  the 
authors  through  literature  reviews  or  personal  interviews 
with  contracting  personnel. 

A  final  limitation  involves  population  inferences  that 
were  made  involving  the  selection  of  only  CONUS  bases  for 
the  quota  sample.  Quota  sampling  was  selected  based  upon 


53 


the  premise  that  sampling  CONUS  bases  would  provide  an 
accurate  representation  for  all  contracting  squadrons. 

toaXyals  ia£-lay.fis.ti~^At±yg  Qy^stigas 

To  adequately  amalyze  the  data  used  throughout  the 
thesis  process,  a  detailed  description  will  be  provided  that 
will  list  each  investigative  question  and  provide  the  means 
by  which  the  question  will  bu  answered. 

Analysis  of  Investigative  Question  Number  One,  what 
performance  factors  currently  exist  within  a  typical 
contracting  squadron? 

The  purpose  of  this  question  was  to  establish  a 
definitive  list  of  all  performance  factors  present  within  a 
typical  contracting  squadron  that  are  capable  of  measuring 
performance.  An  initial  list  of  G3  performance  factors  was 
developed  as  explained  in  the  literature  review  and 
discussions  described  earlier  in  this  chapter.  These 
factors  were  placed  on  the  survey  described  earlier  and 
mailed  to  contracting  personnel  within  the  CONUS.  To  ensure 
the  list  was  definitive,  the  survey  respondents  were  asked 
to  add  any  additional  factors  they  contiidered  important  and 
may  be  useful  in  measuring  organizational  effectiveness. 

Analysis  o£  liive8tiaatlv.a  Question  Number  Two,  which 
of  these  performance  factors  are  considered  significant  for 
measuring  organizational  performance  and  therefore  should  be 
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included  in  the  development  of  an  effective  performance 
evaluation  system? 

The  purpose  of  this  investigative  question  was  to 
evaluate  the  list  of  performance  factors  that  were  assembled 
in  question  one  and  to  analyze  these  factors  by  means  of  a 
survey  questionnaire.  Responses  by  the  sampled  population 
will  provide  data  that  can  be  analyzed  to  determine  the 
importance  of  each  performance  factor  to  be  used  as  a 
performance  measure  for  organizational  effectiveness. 
Analyzing  the  results  received  from  the  survey  responses 
will  determine  which  factors  from  the  definitive  list  of 
performance  factors  were  statistically  significant  for 
measuring  organizational  effectiveness  and  therefore  should 
be  included  in  an  alternative  performance  evaluation  system. 
Using  the  five-point  Likeirt  scale  discussed  earlier,  survey 
respondents  assigned  a  rating  of  Importance  for  each 
performance  factor.  From  these  individual  ratings,  a  mean 
level  of  response  was  calculated  for  each  perfooiance 
factor.  Using  the  mean  level  of  response,  the  performance 
factors  were  then  rank  ordered  from  highest  to  lowest  mean. 
Table  3.1  summarizes  the  large-sample  test  of  hypothesis 
that  was  applied  to  the  rank  ordered  list  to  determine  which 
factors  were  actually  significant  compared  to  the  original 
list  of  63.  An  a  of  .05  and  no“4  was  used  to  determine  the 
most  significant  factors  at  a  95  percent  confidence  level. 
This  approach  in  determining  significance  is  appropriate 
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because  the  purpose  of  the  research  is  to  determine  those 
factors  that  are  most  significant  for  use  in  an  alternative 
performance  evaluation  system.  By  comparing  the  mean  for  a 
performance  factor,  against  somewhat  important, 

only  those  factors  that  are  the  most  significant  will  be 
determined.  This  determination  will  be  made  by  the  test 
statistic  rejecting  the  null  hypothesis  that  the  means  are 
the  same  and  accepting  the  alternate  hypothesis  that  the 
means  are  significantly  different. 

_ _ Table  3.1.  Large  Sample  Test  of  Hypothesis  About  |j, _ 

One-Tailed  Test 

Ho*  P  =  Mo 
Ha*‘  P  <  Mo 
[or  Ha:  M  >Po] 

Test  Statistic; 
z  -  X  -  Po 

Rejection  region:  z  <  -z^ 

_ (or  z  >  z^^  when  Hq;  p  >  mq) _ 

■ejiaiy.sig  o.£.  Iny^stigatlyg  Bvuiiber  Thres-  can  a 

benchmark  be  established  for  each  significant  performance 
factor? 

The  purpose  of  this  investiga  ive  question  was  to 
determine  which  of  the  factors  determined  to  be  significant 
in  invustigative  question  two  are  actually  cap2d3le  of  being 
benchmarked.  As  stated  in  Chapter  II,  factors  to  be 
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benchmarked  must  be  measuradsle,  have  data  readily  available, 
and  provide  useful  information  to  the  organizational 
manager.  Using  these  criteria,  the  list  of  significant 
factors  from  investigative  question  number  two  was  reviewed 
to  determine  which  factors  actually  met  these  criteria.  All 
factors  meeting  those  criteria  can  then  have  benchmarks 
established  for  them. 

Working  with  this  constraint,  the  authors  made  the 
determination  that  in  order  for  a  factor  to  be  considered 
for  the  proposed  benchmark  performance  measure,  the  factor 
must  meet  one  of  two  criteria.  First,  the  factor  must  be 
currently  tracked  and  measured  by  the  monthly  BCAS  report 
prepared  at  each  operational  base,  thus  providing  a  current 
data  source  for  the  benchmarking  methodology.  Second,  if 
the  selected  performance  factor  is  not  currently  measured  by 
BCAS,  then  the  factor  must  have  an  available  data  source 
that  can  be  used  for  the  purpose  of  benchmarking.  An 
example  of  a  performance  factor  in  the  latter  category  is 
that  of  an  IG  rating  received  by  a  contracting  squadron. 
Despite  the  fact  that  IG  ratings  are  not  tracked  by  the  BCAS 
report,  this  factor  could  still  be  used  as  a  benchmark  since 
data  involving  IG  ratings  are  readily  availad^le. 

The  reason  for  developing  a  survey  with  performance 
factors  that  are  not  currently  measured  is  to  identify  all 
potential  factors  that  could  be  useful  in  measuring 
organizational  performance.  One  facet  of  the  research  was 
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to  detezmine  those  factors  that  are  significant  even  if  data 
is  not  currently  being  collected.  Identification  of  these 
significemt  performance  factors  'is  crucial  in  the 
development  of  em  alternative  evaluation  system. 

Analysis  of  Investigative  Question  Number  Four.  Can  a 
viable  performance  evaluation  system,  capable  of  accurately 
measuring  overall  effectiveness  for  a  typical  contracting 
squadron,  be  developed  using  benchmarking  techniques  and 
concepts? 

The  purpose  of  this  question  was  to  determine  if  a 
system  based  upon  benchmarking  techniques  can  provide  a 
viable  alternative  performance  evaluation  system  to  assess 
operational  contracting  squadron  performance.  As  discussed 
in  Chapter  II,  benchmarking  offers  a  comparative  analysis 
among  similar  organisations  so  that  cm  organizational 
manager  is  able  to  assess  the  relative  effectiveness  of  the 
organization.  Additionally,  benchmarking  identifies  those 
organizations  that  are  best-in-clasa  allowing  other 
organizations  to  assess  and  analyze  those  specific 
characteristics  that  led  them  to  achieve  this  success. 

Before  this  comparative  analysis  can  take  place,  the  raw 
data  must  be  collected  to  determine  the  range  for  each 
category  that  is  to  be  benchmarked.  To  answer  this 
question,  the  principles  of  benchmarking  must  be  applied. 
Investigative  questions  two  and  three  have  determined  what 
factors  to  benchmark.  Data  must  now  be  gathered  regarding 
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each  significant  performance  factor  and  used  to  establish  a 
mean,  ia2ixiiQumf  and  minimum  level  of  performance.  These 
performance  levels  should  then  be  presented  to  all 
contracting  squadrons  for  comparison  purposes.  Each 
individual  squadron's  performance  level  could  be  comp&red 
against  a  benchmark  so  that  the  senior  management  element 
could  determine  how  effective  the  squadron  was  performing. 
Squadron  personnel  could  use  this  information  to  determine 
future  work  actions. 

The  actual  benchmarks  were  calculated  from  data  found 
in  the  BCAS  data  report  dated  1  October  1992  for  operational 
contracting  squadrons  within  ACC.  ACC  was  selected  for 
several  reasons.  First  ACC  is  currently  the  largest  command 
in  terms  of  the  number  of  active  operational  bases,  and 
therefore  analysis  of  this  command  would  provide  a  larger 
representative  sample  in  which  to  benchmark.  Second,  ACC 
provided  the  largest  survey  response  rate  and  therefore, 
represents  a  viable  command  in  which  to  provide  benchmarks. 
Finally,  the  Chief  of  Contracting  for  ACC  expressed  interest 
in  the  development  of  an  alternative  performance  evaluation 
system,  and  therefore,  use  of  ACC  for  the  sample  benchmarks 
is  appropriate.  Additionally,  future  research  could  take 
these  findings  and  apply  them  to  all  other  major  commands  to 
determine  if  benchmarking  provides  a  viable  means  for 
measuring  performance  among  operational  contrcicuing 
squadrons . 
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As  of  1  October  1992,  37  operational  contracting 
squadrons  existed  within  ACC,  and  therefore,  all  of  the  data 
gathered  from  the  BCAS  report  for  each  of  these  squadrons 
was  used  for  the  purpose  of  benchmarlcing.  Summaries  of  the 
gathered  data  are  broken  down  into  four  different  categories  • 
labeled  total,  average,  minimum,  and  maximum. 

Total  refers  to  the  summation  of  data  for  the 
particular  performance  factor.  For  instance,  total  for  the 
number  of  delinquent  contracts  would  represent  the  total 
number  of  delinquent  contracts  for  all  of  ACC.  Average 
refers  to  the  overall  average  number  of  delinquent  contracts 
for  the  command.  Average  is  determined  by  simply  dividing 
the  total  figure  by  fhe  number  of  squadrons.  Minimum 
pertains  to  the  smallest  figure  for  each  of  the  factors 
within  the  command.  Using  number  of  delinquent  contracts  as 
an  example,  the  minimum  figure  would  represent  the  least 
number  of  delinquent  contracts  found  for  a  particular 
squadron  within  ACC.  Finally,  maximum  refers  to  the 
greatest  figure  for  each  of  the  factors  found  within  the 
command.  Maximum  for  the  number  of  delinquent  contracts 
would  rapressnt  the  figure  for  the  squadron  with  the 
greatest  number  of  delinquent  contracts.  Consequently,  the 
numbers  presented  for  minimum  or  maximum  can  be  viewed  as 
either  negative  or  positive  depending  upon  which  performance 
factor  is  being  analyzed. 
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Summary 

The  purpose  o£  this  chapter  was  to  present  the 
methodology  used  to  emswer  the  four  investigative  questions 
that  provide  the  focus  for  the  research.  To  answer  the  four 
investigative  questions,  the  appropriate  research  design 
must  be  incorporated. 

For  this  research,  the  research  design  involved  the 
selection  of  performance  factors  found  within  a  contracting 
squadron  that  are  cap2d)le  of  measuring  performance.  Review 
of  applicable  literature  resulted  in  a  list  of  63  factors 
capable  of  measuring  performance  for  a  contracting  squadron. 
Additionally,  research  design  involved  the  selection  of  all 
operational  contracting  squadrons  located  within  the  CONUS 
for  the  population  and  selection  of  the  senior  management 
element  as  the  sample.  Research  design  also  discussed  the 
chosen  instrument  used  in  the  research,  and  perceived 
limitations  associated  with  research  design.  Finally,  the 
chapter  concluded  with  a  discussion  of  the  four 
investigative  questions  that  direct  the  thesis  effort.  The 
outcome  of  this  analysis  is  presented  in  Chapter  IV. 
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IX.  Data  Analysis  and  Findings 


overview 

This  chapter  analyzes  and  evaluates  the  data  received 
from  the  survey  respondents  in  an  attempt  to  present  an 
alternative  performance  evaluation  system  using  benchmarking 
techniques.  First,  the  survey  response  rate  is  addressed, 
followed  by  a  discussion  of  survey  respondent  demographics. 
Next,  each  of  the  four  investigative  questions  are 
addressed.  Analysis  of  investigative  question  one  will 
include  a  list  of  all  performance  factors  present  within  a 
typical  contracting  squadron.  Investigative  question  two 
will  describe  the  procedures  accomplished  to  determine  the 
specific  performance  factors  considered  significant  to  the 
overall  study.  Investigative  question  three  will  discuss 
which  of  these  factors  are  capable  of  being  benclmarked . 
Finally,  investigative  question  four  will  provide  benchmarks 
for  those  factors  identified  in  investigative  question 
three,  followed  by  a  discussion  of  how  a  squadron  commander 
could  effectively  incorporate  the  concepts  of  benchmarking 
to  measure  organizational  performance.. 

amygy  Analysis 

A  total  of  258  surveys  were  mailed  to  all  operational 
contracting  units  within  the  CONUS.  Three  surveys,  to  be 
completed  by  the  Commander,  Deputy  Base  Contracting  Officer, 
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and  Ezecurive  NCO  were  nailed  to  each  contracting  squadron 
listed  in  Appendix  C.  As  of  1  January  1993,  a  total  of  86 
operational  units  were  located  within  the  CONUS,  and  of 
these  86  units,  66  bases  provided  useable  input  from  at 
least  one  respondent.  Of  the  258  surveys  that  were 
adninistered ,  165  (63.95  percent)  of  the  surveys  were 
returned  with  153  (59.3  percent)  of  these  surveys  useable. 
Receiving  a  useable  return  rate  of  59.3  percent  for  a  mail 
survey  can  be  considered  well  above  average  (Emory, 
1991^333).  According  to  Isaac  and  Michael  (1985),  at  least 
59  percent  of  a  sample  should  provide  useable  information 
when  a  study  is  preparing  to  make  a  reliable  inference  about 
a  population  sample  of  258  people.  Overall,  12  surveys  were 
returned  that  were  not  considered  useable.  The  most  common 
reason  for  not  using  a  returned  survey  was  because  the 
respondent  failed  to  answer  all  of  the  survey  questions. 

Table  4.1  summarizes  information  pertai.ning  to  the 
survey  response  rates  by  showing  a  breakdown  by  MAJCOM  and 
total  figures  for  each  category.  The  row  that  is  titled 
Other  includes  all  responses  received  from  coimaands  other 
than  ACC,  AMC,  ATMC,  and  ATC.  Responses  raceix'ed  from  other 
commauds  include  Air  University  and  Air  Force  Space  Command. 
Further,  the  table  provides  information  on  how  many  surveys 
were  sent  out  and  how  many  of  the  returned  surveys  were 
considered  useable. 
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Table  4.1.  Response  Rates 


Commands 

Sent  out 

Used 

%  Used 

ACC 

111 

77 

69.37% 

AMC 

48 

25 

52.08% 

AFMC 

39 

13 

33.33% 

ATC 

39 

23 

58.97% 

Other 

24 

15 

62.50% 

Total 

258 

153 

59.30% 

Demographic  Data 

Part  1  of  the  survey  consisted  of  questions  concerned 
with  gathering  information  on  population  demographics  and  IG 
opinion  questions.  The  purpose  of  asking  demographic 
questions  was  to  receive  pertinent  information  from  the 
sampled  population  in  order  to  assess  the  background, 
experience,  and  related  characteristics  of  the  selected 
sample.  The  results  of  these  questions  can  bo  seen  in  Table 
4.2.  The  column  labeled  Average  Contractiag  Xxperience 
represents  the  average  number  of  years  a  person  has  in 
contracting.  The  final  column  labeled  Avyjrage  Number  of 
Months  represents  the  average  number  of  months  that  the 
respondents  have  spent  in  their  present  job  position. 


Table  4.2.  Demograp] 

t'.ic  Data 

Ccnur^nd 

Numher  of 
Civilians 

Number  of 

Enlisted 

Personnel 

Average 

Contracting 

Fxperience 

Average 

Number  Months 
in  Present 
Position 

ACC 

24 

24 

29 

15.5 

37.29 

AMC 

11 

5 

8 

11.8 

25.96 

AFMC 

4 

7 

2 

15.7 

36.15 

rnimm 

7 

11 

5 

15.4 

41.22 

other 

5 

5 

5 

17.3 

35.07 

Total 

51 

S2  _ 

49 

IS.  10 

35.78 
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The  results  shown  in  Table  4.2  indicate  that  the 


selected  seuaple  has  a  great  deal  of  contracting  experience 
and  contains  evenly  distributed  input  from  officer, 
civilian,  and  enlisted  personnel.  Further,  the  average 
contracting  experience  for  the  respondents  is  15 . 10  years 
followed  by  an  average  of  35.78  months  in  their  present 
position.  Both  of  these  statistics  add  to  the  credibility 
of  the  survey  responses  by  indicating  that  the  respondents 
have  a  vast  amount  of  experience  in  the  areas  in  which  they 
were  providing  feedback. 

Performance  Evaluation  Opinion  Questions.  Survey 
questions  six  through  nine  pertain  to  the  Air  Force  IG 
system  axid  address  those  issues  concerning  development  of  an 
alternative  evaluation  system.  As  discussed  in  Chapter  II , 
the  purpose  of  the  Air  Force  IG  is  to  provide  feedback  to 
commanders  on  the  capability  and  effectiveness  of 
operational  Air  Force  units.  Using  the  rating  criteria  that 
was  discussed  in  Chapter  II,  Table  4.3  summarizes  IG 
inspection  results  as  asked  in  the  survey  and  answered  by 
the  respondents. 

i 


I 

I 

i 
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I 


saw 


Table  4.3.  IG  Information 


Total 


*0“0utstanding,  E-Excellent/  S«Satisfactory, 
M"Marginal,  U»Unsatiafactory,  N/A-Not  Applicable 


The  purpose  of  sununarizing  Tedsle  4.3  was  to  receive  a 
better  understanding  of  the  background  of  the  respondents 
and  to  have  a  batter  assessment  of  the  selected  sample  that 
participated  in  the  research  effort.  By  asking  the 
respondents  what  rating  they  received  on  their  last  IG,  the 
authors  received  an  iiidication  of  what  sample  is  providing 
survey  input  for  analysis.  For  instance,  the  results  of 
Table  4.3  indicate  that  the  occurrence  of  either  an 
Excellent  or  Satisfactory  rating  is  extremely  high,  86 
percent,  while  the  proportion  of  units  rsceiving  either  an 
Outstanding,  Marginal  or  Unsatisfactory  rating  is  very  slim, 
4  percent,  2  percent  and  0  percent,  respectively. 

Tables  4 . 4  and  4 . 5  addz'esses  two  opinion  questions  that 
were  covered  in  the  survey.  The  first  question,  question 
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nujober  eight  on  the  survey,  involves  the  effectiveness  of 
the  current  IG  evaluation  system.  The  question  reads: 
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The  current  coBtaand  IG  evaluation  ayatem  aatlafactorily 
meaaurea  organizational  perfonaance. 

1 .  Strongly  diaagree 

2 .  Diaagree 

3 .  Neutral 

4 .  Agree 

5 .  Strongly  agree 

The  purpose  of  this  question  was  to  solicit  feedback 
from  the  senior  management  element  within  contracting  about 
their  satisfaction  with  the  current  method  of  evaluation. 

If  the  results  of  this  question  Indicate  a  low  response,  any 
value  less  than  3.0,  then  this  input  may  further 
substantiate  the  fact  that  the  current  IG  system  does  not 
satisfactorily  measure  organizational  performance.  In 
addition  to  analyzing  the  results  of  this  one  opinion 
question,  careful  attention  must  be  given  to  the  results  of 
survey  question  number  eight  compared  to  that  of  question 
nuxnber  nine. 

Survey  question  number  nine  solicits  feedback  from  the 

respondents  with  respect  to  whether  an  alternative 

performance  evaluation  system  would  be  useful  in  measuring 

organizational  performance.  The  question  reads: 

Au  evaluation  system,  different  from  the  command  IQ,  would 
be  useful  In  measuring  overall  organizational  performance. 

1.  Strongly  diaagree 

2 .  Disagree 

3 .  Neutral 

4 .  Agree 

5 .  Strongly  agree 
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The  purpose  of  this  question  was  to  determine  whether 
the  respondents  have  a  preference  for  the  development  of  an 
alternative  evaluation  system.  If  the  respondents  indicate 
that  another  evaluation  system  would  be  useful,  the 
creditability  of  the  overall  research  effort  is  increased. 

In  addition,  if  the  mean  response  to  survey  question  number 
eight  is  less  than  3.0,  and  the  mean  response  to  survey 
question  number  nine  is  greater  than  3.0,  then  the  results 
require  further  investigation.  If  the  two  mean  responses 
differ  significantly  from  each  other,  the  conclusion  may  be 
that  the  survey  respondents  are  no  longer  satisfied  with  the 
current  performance  evaluation  system  and  an  alternative 
method  may  be  useful. 

The  summary  of  the  two  opinion  questions  included  in 
the  survey  can  be  found  in  Table  4.4. 
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In  order  to  determine  the  significancs  of  the 
responses,  a  two-tailed  large  sample  test  of  hypothesis  for 
(^1  "  M2)  conducted  at  the  99  percent  confidence  level. 

The  following  tedDle  (Ta±)le  4.5)  shows  the  formula  for 
conducting  this  test . 
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Table  4.5  Large-Sample  Test  of  Hypothesis  for 


Two-Tailttd  Test 
Ho'-  (Pl“M2)“  Do 
Ha 5  Do 

[or  Ha:  (A‘l-M2)>Do] 

where  Dq*  Hypothesized  difference  between  the 
means  which  is  0) 

Test  Statistic: 

z  -  <xi  -  X2)  -  Do 


<xi  -  X2) 


where 

(Xi  -  X2) 


V  oxi2  +  ^^2 

ni  n2 

Rejection  region:  z  <  ••Za/2  or  z  >  Za/2 
where  a/2».005 


Tne  calculated  values  for  the  z-test  are  z»3.45  and 
2tt/2"2.576  with  the  results  of  this  test  rejecting  the  null 

hypothesis  that  the  two  population  means  are  the  same,  and 
therefore  accepting  the  alternative  hypothesis  that  the 
means  are  significantly  different.  This  result  indicates 
that  the  respondents  may  perceive  the  current  IG  evaluation 
method  as  not  adequately  measuring  performance  and  a  new 
performance  evaluation  system  is  needed.  The  results  of 
survey  question  eight  and  nine  help  validate  the  purpose  of 
this  thesis  effort  by  indicating  the  respondents  desire  for 
a  new  performance  evaluation  system. 
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Analysis  of  Investigative  Question  Number  One 

Investigative  question  number  one  reads: 

1.  What  parformauca  factors  currently  exist  within  a 
typical  contracting  squadron? 

The  purpose  of  this  question  was  to  determine  whether  an 
exhaustive  list  of  performance  factors  could  be  tabulated 
and  evaluated  in  a  survey  instrument.  Chapter  III  discussed 
how  the  authors  selected  performance  factors  to  be  included 
on  the  survey  and  what  criteria  was  used  in  the  selection 
process.  Using  this  methodology  the  authors  arrived  at  a 
final  list  of  63  performance  factors  that  was  evaluated  on 
the  survey.  Tedsle  4.o  lists  these  63  factors  found  on  the 
survey  instrument. 


Table  4.6.  List  of  Performance  Factors 


Question 

Number 

Performance  Factor 

10 

Number  of  open  purchase  requests 

11 

Number  of  small  business  goals  achieved 

12 

Last  IG  rating 

13 

Command  or  higher  contracting  awards 

14 

Outstanding  Unit  Citation 

15 

Number  of  contracts  behind  schedule 

16 

Total  number  of  delinquent  contractors 

17 

Number  of  SF  129  pacXages  mailed 

18 

Number  of  new  vendors  identified  and  loaded 
in  BCAS 

20 

Number  of  STEP  promotions 

21 

Number  of  protests  received 

22 

Number  of  protest  successfully  defended 

23 

Total  number  of  ratifications 

24 

Percentage  of  competitive  actions 

26 

Total  number  of  contracting  actions 

27 

Total  number  of  centralized  actions 

28 

Total  number  of  decentralized  actions 

29 

Total  dollars  awarded  competitively 

30 

Number  of  interest  payments  paid  due  to  late 
processing  of  contractor  invoice 
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Table  4.6.  Continued 


Parfonance  Factor 


Niunber  of  contracts  awarded  using  source 
selection 

Number  of  undefinitized  actions 
Number  of  BPAs  administered 

Number  of  amendments  due  to  contracting  error 
Niunber  of  customer  education  classes  provided 
Number  of  man'-hours  expended  to  support 
deployments ,  etc . 

Number  of  in-house  proficiency  training 
sessions 

Number  of  squadron  Operating  Instructions 
NuiDber  of  self  inspections  conducted 
Number  of  Value  Engineering  Change  Proposals 
received 

Number  of  Value  Engineering  Change  Proposals 
approved 

Percent  of  time  BCAS  is  available  to  users 
Nuiober  of  purchase  requests  over  90  days 
Number  of  purchase  requests  over  120  days 
Number  of  vendor  follow-ups  due  to  late 
delivery 

Number  of  walk-through  purchase  requests 
Total  nufJaer  of  line  items  received  priority 
1-8 

Total  number  of  line  items  received  all 
priorities 

Number  of  different  customer  organizations 

served  by  contracting 

Total  dollars  awarded 

Total  number  of  modifications 

Priority  1-3  CALT 

?  ‘iority  4-8  CALT 

Priority  9-15  CALT 

The  overall  average  CALT 

rc  tal  nuriber  of  Non-Appropriated  Fund  actions 

Total  Non-Appropriated  Fund  dollars  awarded 

Total  number  of  Section  8A  actions 

Total  Section  8A  dollars  awarded 

Total  number  of  line  It ^ms  awarded 

Total  number  of  centralized  line  items 

^.’.warded 

Total  number  of  decentralized  line  items 
awarded 

Total  centralized  .  ..ars  awarded 
Total  decentralized  dollars  awarded 
Total  set-asj.do  actions  awarded 


Table  4.6.  Continued 


Question 

Hunber 

Perfomance  Factor 

34 

Total  number  of  large  business  actions 
awarded  that  were  availj&ble  for  small 
business 

85 

Total  large  business  dollars  awarded  that 
were  available  for  small  business 

86 

Ratio  of  small  business  dollars  awarded 
divided  by  total  dollars  available  for  small 
business 

87 

Percentage  competitive  dollars  measure 

88 

Total  number  of  active  contracts 

89 

Total  number  of  active  A&E  and  construction 
contracts 

SO 

Total  number  of  active  service  contracts 

91 

Total  number  of  commodities  contracts 

In  addition  to  the  63  performeuice  factors  that  were 
considered  on  the  survey  the  authors  also  analyzed  the 
write-in  responses  recorded  in  Part  IV  of  the  survey.  The 
purpose  of  this  part  of  the  suzrvey  was  to  solicit  feedback 
fron  the  respondents  on  any  performance  factor  that  nay  have 
been  missing  from  the  list  found  on  the  survey.  Of  the  166 
surveys  that  were  received,  81  (49  percent),  included  at 
least  one  write'-in  response.  These  responses  are  shown  in 
Table  4.7. 


Teible  4.7.  Write-in  Responses 


tfrite«*in  response 

Husdber  of  responses 

Customer  support  and 

32 

satisfaction 

Recognition  programs 

4 

Customer  support  lead  time 

4 

Additional  duties 

4 

Customer  training 

4 

TQM  training 

4 

Contingency  procurement  and 
trained  personnel 

3 

DD  350  errors 

2 

Morale 

2 

Employees  award  program 

1 

Adequate  facilities  and 
parJcii^ 

1 

QA£  trained 

1 

Total  QAE  appointed 

1 

ifW7. 1  •  .V .  f  ^  A  Mr  M  l  rTTwnrarr.^MiiiMi 

1 

1  Personnel  turnover 

1 

Environmental  contracts 

1 

Disciplinary  actions 

1 

Number  of  conOTessionals 

1 

Number  of  solicitatior.  review 
comments 

1 

Number  of  contract  review 

1 

comments 

Number  of  modification  review 
comments 

1 

Number  of  legal  review 
comments 

1 

Discrepancies  per  order 

1 

1 

Analysis  of  the  35  writs-in  rssponsss  along  with  the  63 
original  performance  factors  indicates  that  there  are 
potentially  88  performance  factors  that  may  be  used  to  the 
measure  the  performance  of  an  operational  contracting 
squadron.  These  88  factors  will  be  further  analyzed  in 
investigative  question  two  to  determine  which  of  these 
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factors  are  considered  significant  and  should  be  included  in 
an  alternative  performance  eva?  ation  system. 

Analysis  of  Investigative  Question  Number  Two 

This  portion  of  the  chapter  discusses  investigative 
question  number  two  which  reads: 

2.  Nhich  of  those  perfomance  factors  are  considered 
significant  fcr  sMasuring  organizational  performance  and 
therefore  should  be  included  in  the  development  of  an 
effective  performance  evaluation  system? 

The  purpose  of  this  question  was  to  analyze  the  list  of 
63  perfonoeuace  factors  found  on  the  survey  along  with  emy  of 
the  factors  that  were  included  as  a  write-in  response,  amd 
determine  a  final  list  of  significant  performance  factors 
that  could  be  used  in  the  development  of  a  performance 
evaluation  system. 

To  determine  the  significant  performance  factors  among 
the  write-in  responses,  descriptive  statistics  will  be 
employed.  Through  the  use  of  this  test  only  one  response, 
customer  support  and  satisfaction,  is  significant.  This 
particular  response  generated  input  from  32  (20.92  percent) 
of  the  survey  respondents,  while  the  remaining  vrite-in 
responses  received  less  than  two  percent  input  from  the 
respondents  and  therefore,  none  of  the  remaining  write-in 
responses  will  be  considered  significant  in  this  study. 

For  the  remaininc  63  p  ^rformance  factors,  determination 
of  significance  involved  a  two-step  process.  First,  the 
factors  were  sorted  by  rank  order  which  was  determined  by 
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the  calculated  mean  for  each  of  the  factors.  After 
determining  the  rank  order  of  the  factors,  the  second  step 
involved  determining  which  of  the  factors  are  most 
significant  and  should  ^  ^  included  in  a  benchmarking 
evaluation  s^^stem.  This  step  involved  conducting  a  test  of 
hypothesis  that  compares  the  means  of  two  populations  to 
provide  input  on  which  factors  are  most  significant  in 
relative  importance.  This  process  will  be  discussed  in 
detail  later  in  the  chapter. 

Table  4.8  presents  a  summary  of  each  of  ihese 
performance  factors  ranked  from  most  important  to  least 
important.  The  column  titled  mean  level  of  xnportance 
represents  the  average  responsu  for  the  153  survey  inputs 
and  this  figure  determines  the  final  rank  order  for  each 
performance  factor.  The  calculated  averages  are  based  on 
the  scale  of  importance  used  on  the  survey — (1)  Definitely 
not  important;  (2)  Somewhat  not  important;  (3)  Neutral;  (4) 
Somewhat  important;  ana  (5)  Definitely  important.  Appendix 
D  contains  a  table  that  lists  the  63  performance  factors  in 
rank  order  along  with  average  ratings  for  each  of  the 
MAJCCMs  and  ACC  Staff.  The  reason  for  including  this 
appendix  is  to  present  the  differences  that  exist  among 
commands  in  determining  the  importance  of  performance 
factors. 
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Table  4.8.  Rank  Order  of  Performance  Factorb 


Perfonaance  factor  Survey  Mean  level  Rank 

question  of  j  order 

number  imoortance  | 


46  4.699 


4.536 


11 


Number  of 
contracts  behind 
schedule 


Percent  of  txme 
BCAS  is  avail<d>le 
to  users 


Numbex  of  man-* 
hours  expended  to 
support 

deployment s,  etc. 
Number  of 
amendments  due  to 
contracting  error 
Total  nuiater  of 
line  items 
received  all 
priorities 


Total  number  of 
active  service 
contracts 


Total  mimber  of 
active  contracts 


Number  of  customer 
education  classes 
provided  _________ 

Number  of  "px'otests 

successfully 

defended 


Total  number  of 
line  item^ 
received  priority 

1-8 _ 

Total  number  of 
active  A&E  and 
construction 
contracts 


Total  number  of 

commodities 

contracts 


t88 


4.503 


4.484 


4.451 


4,410 


3 


4.366 


4.359 


4.317 


j  4.2?5 

i 

13 

.  .. 

4.24R 

14 
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proficiency 
traininq  sessions 


Nuiaber  of  open, 
purchase  requests 


Total  number  of 

delinquent 

contractors 


Nuiober  of  xnterest 
payments  paid  due 
to  late  processing 
of  contractor 
invoice 


Total  number  of 

contracting 

actions 


Percentage  of 

competitive 

actions 


Total  number  of 
modifications 


Number  of  vendor 
follow-ups  duo  to 
late  delivery 


Total  dollars 
awarded 
competitive! 


Number  of 

different  customer 
organizations 
served  by 
contract in 


Total  number  of 
ratifications 


1-3  CALT 


Total  number  of 
line  items  awarded 


Total  number  of 

centralized 

actions 


Total  number  of 
centralized  line 
items  awa-ded 


Number  of  walk¬ 
through  purchase 

requests _ 

Percentage 
competitive 
dollars  measui'e 


4.222 


4.190 


4.190 


4.176 


4.163 


4.144 


4.137 


4.124 


4.059 


4.039 


4.026 


4.013 


4.013 


4,007 


3.980 


3.980 


3,954 


Table  4.8 


Total  nimber  of 
large  business 
actions  awarded 
that  were 
available  for 
small  business 


Number  of  self 

inspections 

conducted 


Total  large 
business  dollars 
awarded  that  were 
available  for 
small  business 


Number  of  Reports 
of  Discrepancy 


The  priority  9-15 
CALT 


Number  of  protests 
received 


undefinitized 

actions 


Total  Section  8A 
dollars  awarded 


Total  set-'aside 
actions  awarded 


Ratro  of  small 
business  dollars 
awarded  divided  by 
total  dollars 
available  for 
small  business 


Total  number  of 

decentralized 

actions 


Total  centralized 
dollars  awarded 


Number  of  small 
business  goals 
achieved 


Total  dollars 
awarded 


Total  number  of 
Section  8A  actions 


Number  of  BPAs 
administered 


.  Continued 


84 

3.928 

41 

j 

3.889 

85 

3.882 

19 

3.869 

69 

3.869 

68 

3.830 

21 

3.817 

32 

3.814 

73 

3.778 

3.771 


3.768 


3.765 


3.758 


3.758 


3.758 


3.641 


9 


Total  number  of 
decentralized  line 
items  awarded 

3.608 

50 

Outstanding  Unit 
Citation 

14 

3.490 

51 

Total 

decentralized 
dollars  awarded 

78 

3.477 

52 

Number  of 
contracts  awarded 
using  source 
selection 

31 

3.431 

53 

Command  or  higher 
contracting  awards 

13 

3.408 

54 

wmamBsssmam 

12 

3.350 

55 

Number  of  STEF 
promotions 

20 

3.105 

56 

Total  number  of 
Non-Appropriated 
Fund  actions 

70 

2.958 

57 

Total  Non- 
Approprlated  Fund 
dollars  awarded 

71 

2.944 

58 

Numbe  of  Value 
Engineering  Change 
Proposals  approved 

43 

2.876 

59 

Number  of  new 
vendors  identified 
and  loaded  in  BCAS 

18 

2.824 

60 

Number  of  squadron 

Operating 

Instructions 

40 

2.797 

61 

Number  of  Value 
Engineering  Change 
Proposals  received 

42 

2.784 

62 

Number  of  SF  129 
packages  mailed 

17 

2.601 

63 

Customer  support 
and  satisfaction 

N/A 

N/A 

In  order  to  sufficiently  Include  e\ll  factors  that  are 
deemed  significant  based  on  Input  from  the  survey 
respondents,  a  test  of  hypothesis  was  conducted  to  determine 
which  factors  are  most  significant  by  comparing  the 
population  means  for  each  performance  factor  against  a 
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neutral  response.  The  test  involved  a  large  sample  one-tail 
z-test  with  an  a  of  .05  to  determine  which  factors  are 
significantly  important  and  should  be  included  ii:  a  new 
performance  evaluation  system.  The  population  mean  that  is 
under  consideration  involves  the  mean  that  was  calculated 
for  each  individual  performance  factor  against  the  response 
of  4,  somewhat  important,  on  the  Likert  scale.  The  reason 
for  this  selection  was  based  upon  the  fact  the  overall  mean 
for  all  performance  factors  was  found  to  be  3.88  and 
therefore  running  the  test  of  hypothesis  found  in  Table  4.9 
against  a  of  4  would  provide  results  of  the  must 
significant  factors  from  the  original  list.  Table  4.9 
contains  the  test  of  hypothesis  that  was  used  to  determine 
the  signific2mt  performance  factors. 

_ Table  4.9.  Large  Sample  Test  of  Hypothesis  About  ^ _ 

One-Tailed  Test 

HoJ  •  ^^o 
KaJ 

[or  fx  >Aio] 

Test  Statistic: 

z  -  X  -  Mo 

Rejection  region:  z  <  -z^ 

(or  z  >  Ztt  when  Ha;  M  >  mo) 


» 

At  the  95  percent  confidence  level,  any  factor  that 
rejects  the  null  hypothesis  and  accepts  the  alternative 
hypothesis  is  considered  to  be  a  significant  factor.  The 
results  of  this  test  indicate  that  22  factors  are 


significantly  different  from  4.0-  Additionally,  the  write- 


in  response,  customer  support  and  satisfaction,  was  also 
considered  significant  as  described  earlier,  and  therefore, 
a  total  of  23  factors  were  considered  significant  from  the 
original  list  of  factors.  Appendix  £  contains  the 
methodology  results  of  the  z-test.  Table  4.10  contains  the 
23  significant  performance  factors  in  rank  order  from  most 
important  to  least  important  based  upon  the  calculated 


averages  shown  in  Table  4.8. 


Table  ..10.  Significant  Performance  Factors  at  the  95% 

_ _ Confidence  Level _ 

Question  Perfommnce  Factor  Rank  Order 

Number 


46 

45 

15 

44 

37 


35 

62 


90 

88 

36 

22 

61 

89 


91 


Number  of  pux'chase  requests  over  120  days 
Number  of  purchase  requests  over  90  days 
Number  of  contracts  behind  schedule 
Percent  of  time  BCAS  is  available  to  users 
Number  Of  man-hours  expended  to  support 
deployments,  etc. 

Number  of  amendments  due  to  contracting  error 
Total  nusUaer  of  line  items  received  all 
priorities 

Total  number  of  active  service  contracts 
Total  number  of  active  contracts 
Number  of  customer  education  classes  provided  10 


Number  of  protests  successfully  defended  11 

Total  number  of  line  items  received 
priority  1-8  12 

Total  number  of  active  A&E  and  construction 
contracts  13 

Total  ntimber  of  commodities  contracts  14 


82 


invo  r-oow 


Table  4.10.  Continued 


Number 

39  Number  of  in-house  proficiency  training 

sessions  15 
10  Number  of  open  purchase  requests  16 
16  Total  number  of  delinquent  contractors  17 
30  Number  of  Interest  payments  paid  due  to  late 

processing  of  contractor  invoice  18 
26  Total  number  of  contracting  actions  19 
24  Percentage  of  competitive  actions  20 
65  Total  nuiDber  of  modifications  21 
47  Number  of  vendor  follow-ups  due  to  late 

delivery  22 
N/A  Customer  support  and  satisfaction  * 
*  based  on  input  from  write-in  response _ 


The  23  factors  found  in  Table  4.10  provide  for  a  list 
of  performance  factors  that  can  be  considered  for 
investigative  question  three. 


Analysis  of  Investigative 

Investigative  question  ntuaber  three  reads: 

3.  Can  a  benchmark  be  eatablisbad  for  each  significant 
performance  factor? 

Despite  the  fact  that  a  list  of  23  factors  were 
considered  statistically  significant,  benchmarks  could  not 
be  established  for  all  of  them.  In  order  for  benchmarking 
to  be  adequately  developed  the  specific  factors  should 
possess  measures  that  are  readily  avail£d>le.  Working  with 
this  constraint,  the  authors  made  the  determination  that  in 
order  for  a  factor  to  be  considered  for  the  proposed 
benchmark  performance  measure,  the  factor  must  meet  one  of 
two  criteria.  First,  the  factor  must  be  currently  tracked 
and  measured  by  the  monthly  BCAS  report  prepared  at  each 
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operational  base,  thus  providing  a  current  data  source  for 
the  benchmarking  methodology.  Second,  if  the  selected 
performance  factor  is  not  currently  measured  by  BCAS,  then 
the  factor  must  have  an  avail£d}le  data  source  that  can  be 
used  for  the  purpose  of  benchmarking.  Appendix  F  lists 
those  factors  that  are  considered  significant,  yet  are  not 
currently  measured.  In  Chapter  V,  under  future 
recononendations ,  this  area  will  be  discussed  more 
thoroughly. 

Utilizing  the  two  steps  discussed  above,  the  previous 
list  of  23  significant  factors  was  reduced  to  a  'final  list 
of  10.  Table  4.11  contains  the  list  of  factors  that  will  be 
used  for  the  benctunarking  methodology  in  appropriate  rank 
order  as  determined  by  the  mean  level  of  importance. 


Table  4.11.  Rank  Order  of  Performemce  Factors 


Considered  for  Benchmarking 


Performance 

factor 

Survey 

Question 

Number 

Mean 

Level  of 
Import an 
ce 

Rank  order 

Number  of 
contracts 
behind 
schedule 

15 

4.503 

1 

Total  number 
of  line  items 
received  all 
priorities 

62 

4.37" 

2 

Total  number 
of  active 
service 
contracts 

90 

4.366 

3 

Total  number 
of  active 
contracts 

88 

4.359 

4 
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Tal 

oIb  4.11.  Continued 

Total  number 
of  line  items 
received 
priority  1-8 

61 

4.307 

5 

Total  number 
of  active  A&E 
and 

construction 

contracts 

89 

4.275 

6 

Total  number 
of 

commodities 

contracts 

91 

4.248 

7 

Total  number 
of 

contracting 

actions 

26 

4.163 

8 

Percentage  of 

competitive 

actions 

24 

4.144 

9 

Total  niimber 
of 

modifications 

65 

4.137 

10 

1  ■  ■■  - _ 

Analysis  of  Invastiaatlve  Quastion  Nmober  Four 

In  order  to  utilize  benchmarking  techniques  as  a  means 
to  measure  performance,  investigative  question  number  four 
must  be  analyzed  and  discussed.  Investigative  question 
number  four  reads: 

4.  Can  a  viable  performance  evaluation  system,  capable  of 
accurately  measuring  overall  effectiveness  for  a  typical 
contracting  squadron,  be  developed  using  benchmarking 
techniques  and  concepts? 

As  discussed  in  Chapter  II,  benchmarking  offers  a 
comparative  analysis  among  similar  organizations  so  that  an 
organizational  manager  is  able  to  assess  the  relative 
effectiveness  of  the  organization.  Additionally, 
benchmarking  identifies  those  organizations  that  are  best- 


in-class  allowing  other  organizations  to  assess  and  analyze 


those  specific  characteristics  that  led  them  to  achieve  this 
success.  Before  this  comparative  analysis  can  take  place, 
the  raw  data  must  be  collected  to  determine  the  range  for 
each  category  that  is  to  be  benchmarked. 

Tables  4.12  through  4.21  contain  benchmarks  for  the  10 
performance  factors  identified  in  Table  4.11.  The  tables 
contain  the  benchmark  as  calculated  from  data  fourd  in  the 
BCAS  data  report  dated  1  October  1992  for  operational 
contracting  squadrons  within  ACC.  As  of  1  October  1992,  37 
operational  contracting  squadrons  existed  within  ACC,  and 
therefore,  all  of  the  data  gathered  from  the  BCAS  report  for 
each  of  these  squadrons  was  used  for  the  purpose  of 
benchmarking.  Summaries  of  the  gathered  data  are  broken 
dpwn  into  four  different  categories  labeled  total,  average, 
minimum,  and  maximiun.  Presentation  of  the  raw  data  for  each 
of  the  10  performance  factors  and  all  of  the  ACC  squadrons 
can  be  seen  in  Appendix  G. 

Table  4.12  pertains  to  question  number  15 — inumber  of 
contracts  behind  schedule.  The  benchmarking  summary  for 
this  factor  is  shown  below. 


Tab le  4.12.  Analysis  of  Question  15 


Total 

849 

Average 

22.95 

Minimum 

0 

Maximum 

205 

86 


Table  4.13  covers  question  number  62 — total  number  of 


line  items  received  all  priorities.  The  benchmarking 
summary  for  this  factor  is  shown  below. 


T^le  4.13.  Analysis  of  Question  62 


1  Total 

726,271 

wmssmmmm 

19,628.95 

Minimum 

1,005 

Maximuun 

44,796 

Table  4.14  refers  to  survey  question  number  90 — total 
number  of  active  service  contracts.  The  summary  for  this 
perfoi-mance  factor  is  shown  below. 


Table  4.14.  Analysis  of  Question  90 


Total 

2,381 

64.35 

Minimum 

13 

Maximum 

159 

Table  4,15  covers  question  number  88  on  the  survey — 
total  number  of  active  contracts.  The  figures  for  this 
performance  actor  are  shown  below. 


Table  4.15.  Analysis  of  Question  80 


1  Total  ! 

5,401 

145.97 

Minimum 

22 

Maximum 

442 

Table  4.16  pertains  to  question  number  61 — total  number 


of  line  items  received  priority  1-8.  The  results  of  this 
performance  factor  are  shown  below. 


T^le  4.16.  Analysis  of  Question  61 


Total 

193,403 

Average 

5,227.11 

Hinim\un 

0 

Maximum 

16,053 

Table  4.17  refers  to  survey  question  number  89 — total 
number  of  actiye  and  construction  contracts.  The 
figures  for  this  performance  factor  are  shown  below. 


Table  4.17.  Analysis  of  Question  89 


Total 

2,161 

58.41 

Minimum 

0 

Maximum 

210 

Table  4.18  covers  question  number  91  on  the  survey — 
total  number  of  commodities  contracts.  The  benchmarking 
summary  for  this  performance  factor  is  shown  below. 


Table  4.18.  Analysis  of  Question  93 


Total 

715 

Average 

19.32 

Minimum 

1 

Maximum 

72 

Table  4.19  ref ere  to  question  number  26  on  the  survey 
total  number  of  contracting  actions.  The  figures  for  this 
performance  factor  are  shown  below. 


Table  4.19.  Analysis  of  Question  26 


Total 

1,060,231 

28,654.89 

Minimiam 

1,875 

Maximum 

92,658 

Table  4.20  covers  survey  question  number  24 — percentage 
of  competitive  actions.  Results  of  this  category  are  shown 
below. 


Table  4.20.  Analysis  of  Question  24 


Total 

N/A 

Average 

97.67% 

Minimum 

69.80% 

Meocimum 

100% 

Teible  4.21  pertains  to  question  number  65 — total  number 
of  modifications  The  results  for  thJ.s  performance  factor 
are  shown  below. 


Table  4.21.  Analysis  of  Question  65 


Total 

19,969 

Average 

1,350.51 

Minimum 

492 

Maximum 

3,510 

The  10  benchmarks  provided  in  tables  4.12  through  4.21 
present  data  that  can  be  used  by  a  senior  management  element 
of  a  contracting  squadron  to  gauge  performance.  The 
benchmarks  provide  data  for  the  10  most  significant 
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performance  factors  as  rated  by  the  survey  respondents  that 
are  currently  measured  in  BCAS  reports.  These  benchmark 
summaries  provide  feedback  for  an  organization  and  permit  an 
organization  to  compare  their  current  performance  against 
the  performance  of  superior  contracting  squadrons  within 
ACC.  For  instance,  survey  question  number  15  pertains  to 
the  total  number  of  contracts  behind  schedule.  For  this 
category  the  number  of  contracts  behind  schedule  range  from 
0  to  205,  with  an  average  of  22.95.  Using  this  performance 
factor  as  a  means  in  which  to  evaluate  performance  would 
indicate  that  in  order  for  an  organization  to  achieve  ACC 
best  standards,  a  contracting  squadron  must  have  0  contracts 
behind  schedule.  In  order  to  effectively  incorporate 
benchmarking,  if  a  particular  squadron  was  not  operating 
effectively  in  this  area,  the  squadron  would  then  attempt  to 
incorporate  the  methods  employed  by  the  squadron  operating 
most  effectively. 

Samraary 

The  purpose  of  this  chapter  was  to  present  the  findings 
obtained  from  data  analysis  performed  as  part  of  thxs 
research  effort.  The  authors  began  the  chapter  by 
presentiivg  the  survey  response  rate  followed  by  a  discussion 
of  population  demographics.  Receiving  a  response  rate  of 
59.3  percent  along  with  evenly  distributed  responses  among 
the  officer,  civilian,  and  enlisted  respondents  adds  to  the 
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validity  of  the  research  analysis.  Furthermore,  the 
respondents  appear  to  have  the  applicable  knowledge  to 
answer  the  survey  questions  since  the  average  respondent  has 
15.10  years  in  contracting  followed  by  35.78  months  in  his 
or  her  present  position.  In  addition,  this  section  of  the 
chapter  covered  the  opinion  questions  and  responses  received 
from  the  survey  respondents.  The  findings  for  this  section 
of  the  chapter  indicate  that  the  average  respondent  is  not 
satisfied  with  the  current  IG  inspection  system,  and  would 
be  interested  in  having  an  alternative  system  developed. 

Finally,  the  majority  of  the  chapter  focused  on 
analysis  of  investigative  questions  one  through  four  in  an 
attempt  to  present  an  alternative  performance  evaluation 
system  usixig  benchmarking  techniques.  The  findings  for  the 
first  investigative  question  indicate  that  potentially  88 
performance  factors  exist  that  may  be  capable  of  measuring 
perfoimance  for  a  contracting  squadron.  The  88  performance 
factors  are  comprised  of  the  63  performance  factors  included 
on  the  survey,  and  25  write-in  responses. 

Findings  for  the  second  investigative  question  include 
the  rank  order  for  the  63  performance  factors  indicating 
order  of  importance  for  each  of  the  factors.  The  rank  order 
was  determined  by  calculating  the  mean  responses  for  each  of 
the  factors.  DoJtermining  which  factors  were  significant 
involved  a  large  sample,  one-tail  z-test  at  the  95  percent 
confidence  interval,  with  the  results  indicating  22  factors 
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to  be  significantly  different,  and  one  factor  included  as 
the  result  of  write-in  responses. 

The  third  investigative  question  involved  daterroining 
which  performance  factors  met  the  benchmarking  criteria  of 
measurable,  available,  and  important.  Ten  performance 
factors  met  this  criteria  and  formed  the  basis  for  a 
performance  evaluation  system  based  upon  benchmarking 
principles. 

The  final  investigative  question  involved  the 
calculation  of  benchmarks  for  each  of  the  10  significant 
factors  in  an  effort  to  develop  an  alternative  evaluation 
system,  using  the  benchmarking  steps  discussed  in  Chapter 
II,  and  the  data  collected  from  the  BCAS  report,  benchmarks 
for  these  10  performance  factors  were  calculated  and 
provided  in  tables  4.12  through  4.21.  These  benchmarks  can 
then  be  used  by  the  senior  management  element  of  a 
contracting  squadron  to  gauge  performance  and  provide 
feedback  for  the  organization. 
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Y*  Conclusions  and  Recommendations 


Overview 

This  chapter  presents  the  conclusions  drawn  from  the 
analysis  of  the  investigative  questions  and  addresses  the 
overall  research  objective.  Limitations  to  this  research 
effort  as  well  as  recommendations  for  future  efforts  are 
also  provided. 

Specific  Conclusions 

The  purpose  of  this  thesis  effort  was  to  present  an 

I 

alternative  performance  evaluation  system  capable  of 
measuring  performance  within  a  typical  operational 
contracting  squadron.  To  assess  organizational  performance, 
an  effective  performance  evaluation  system  must  be 
available,  and  it  should  be  established  based  upon  valid  and 
reliable  performance  measures.  Furthermore,  these  measures 
should  be  considered  important  by  the  individuals  developing 
the  performance  evaluation  system,  and  the  system  should  be 
established  by  the  individuals  whose  performance  will  be 
evaluated  by  the  system.  B'inally,  this  system  should  be 
tapered  to  the  environment  in  which  it  is  to  be  used. 
Specific  conclusions  from  this  research  effort  are  discussed 
below: 

1.  The  current  system  of  evaluation  used  to  assess 
contracting  squadron  performance  is  based  upon  traditional 
measures  and  is  lacking  certain  criteria  identified  as 
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1 

I 


essential  for  an  effective  performance  evaluation  system. 

An  alternative  evaluation  system  should  be  developed  for 
assessment  of  operational  contracting  squadrons. 

Benchmarking  techniques  can  be  applied  in  the  development  of 
this  system  as  a  capable  means  of  measuring  squadron 
performance. 

2.  Based  on  a  review  of  current  literature, 
interviews,  and  feedback  from  the  survey  instrument,  88 
performance  factors,  capable  of  measuring  organizational 
performance,  exist  within  a  typical  contracting  squadron. 

Of  these,  24  were  considered  statistically  significant  and 
analyzed  for  benchmarking  analysis.  Potential  candidates 
for  benchmarking  are  those  factors  that  are:  (1)  measurable, 
(2)  have  data  readily  available,  and  (3)  provide  useful 
information  to  the  organizational  management.  Applying  this 
criteria  to  the  list  of  24  factors,  benchmarks  were  provided 
for  10  of  the  factors. 

3.  Many  of  the  factors  currently  used  by  the  Air  Force 
to  measure  performance  were  not  considered  significant  by 
the  survey  respondents  for  measuring  performance.  Among 
those  factors  were  IG  ratings,  receipt  of  Outstanding  Unit 
Citation,  and  squadron  awards  received  from  MAJCOM  and 
higher  agencies. 

4 .  Benchmarking  can  be  considered  us( f ul  in  the 
development  of  an  alternative  performance  evaluation  system. 
It  provides  information  on  how  a  particular  organization  is 
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perf orraing  with  respect  to  similar  organizations,  along  with 
identifying  those  organizations  that  are  considered  best-in¬ 
class.  Providing  this  feedback  allows  an  organizational 
manager  to  study  best-in-class  practices  and  implement  those 
procedures  in  an  effort  to  improve  performance.  Finally, 
benchmarking  provides  an  objective  and  viable  means  of 
evaluating  performance  in  a  changing  environment. 

Limitations 

Several  different  and  specific  limitations  exist  in  the 
overall  research  effort.  These  limitations  should  not 
negate  the  results  of  the  study  but  should  be  considered  by 
researchers  contemplating  follow-on  research. 

1.  The  list  of  88  performance  factors  may  not  be  all- 
inclusive  for  measuring  performance  within  an  operational 
contracting  squadron.  Since  35  write-in  responses  were 
added  to  the  survey,  the  indication  is  that  further 
performance  factors  may  exist  within  a  squadron. 

2.  Only  10  out  of  the  24  significant  factors  could  be 
considered  for  the  purpose  of  benchmarking  due  to  data 
availability.  The  remaining  14  factors  listed  in  Appendix  F 
shoull  be  considered  for  benchmarking  analysis.  To 
accomplish  this,  a  method  to  track  and  gather  data  must  be 
established. 

3.  Despite  the  statistical  significance  of  the  10 
performance  factors  that  were  used  for  the  benchmarking 
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analysis,  not  all  o£  these  factors  appear  capable  of 
accurately  measuring  performance  within  a  contracting 
squadron.  In  order  to  provide  more  useful  measures  for  a 
contracting  squadron,  ratios  should  be  developed  for  the 
significant  factors. 

4.  This  study  was  concerned  with  CONUS  contracting 
squadrons  only,  therefore  the  results  cannot  be  assumed  to 
be  valid  for  overseas  contracting  squadrons. 

5.  The  benchmarks  established  during  this  study  are 
only  as  accurate  as  the  BCAS  data  used  to  establish  them. 

Specific  recommendations  offered  for  consideration  as  a 
result  of  this  study  are  as  follows, 

1.  The  Air  Force  contracting  community  should 
determine  a  definitive  list  of  performance. factors  that  are 
important  and  can  be  used  in  the  development  of  an 
alternative  performance  evaluation  system.  The  relative 
importance  of  each  performance  factor  should  be  analyzed  and 
compared  by  command  to  determine  if  significant  differences 
exist.  Differences  should  be  identified  and  reconciled 
prior  to  establishing  an  alternative  performance  evaluation 
system. 

2.  Periodic  reviews  for  each  performance  factor  should 
be  conducted.  Additionally,  established  benchmarks  should 
be  continuously  reviewed  and  recalibrated  as  necessary. 
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3.  Implementation  of  benchmarking  techniques  should  be 
internalized  by  senior  management  throughout  the  Air  Force 
contracting  community. 

Specific  recommendations  for  future  research  include 
the  following. 

1.  Continue  development  of  an  alternative  performance 
evaluation  system  that  utilizes  benchmarking  techniques  and 
incorporates  those  characteristics  needed  to  evaluate 
performance  in  a  total  quality  environment.  The  selected 
system  should  enable  a  squadron  commander  to  strictly  focus 
on  the  mission  of  the  squadron,  whereby  the  performance 
evaluation  system  would  provide  an  objective  measure  of 
mission  accomplishment. 

1.  Establish  ratios  for  all  benchmarks  so  that  the 
benchmarks  can  be  used  as  a  standardized  measure  of 
performance. 

2.  Establish  a  method  to  measure  those  factors  found 
significant  but  not  currently  tracked  by  EGAS  or  other 
sources.  Benchmarks  could  then  be  established  for  these 
factors,  thereby  providing  squadron  commanders  a  more  in- 
depth  system  to  evaluate  contracting  squadron  performance. 
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DEPARTMENT  OF  THE  AIR  FORCE 

HEAOOUASTERS  AiR  COMBAT  COMMAND 
l/NOLEY  AIR  FORCE  BASE.  VIRGINIA 


HQ  ACC/LGC 

130  Douglas  Street,  Suite  210 
Langley  AFB  VA  23665-2791 

Captain  Mark  Fahrenkamp 
AFIT/LSG 

Wright  Patterson  AFB  OH  45433 
Dear  Captain  Fahrenkatap 

As  you  search  for  a  topic  for  your  thesis,  I  would  like  to 
suggest  an  item  of  concern.  Air  Combat  Command  (ACC)  is 
interested  in  establishing  a  metric  or  metrics  to  determine  the 
effectiveness  of  a  given  contracting  squadron's  support  to  the 
wings'  mission  needs.  Such  a  metric(s}  would  be  beneficial  not 
only  as  an  assessment  by  ACC  but  also  to  squadron  commanders  as 
they  strive  to  identify  areas  which  may  need  process  improvements. 

I  offer  the  support  of  the  ACC/LGC  staff  to  assist  you  further 
with  problem  identification  or  research  materials.  If  you  would 
like  to  discuss  this,  contact  Mr  Carpenter,  LGCP  at  DSN  574-5096. 

Sincerely 


ROBERT  A.  CORDANO,  Colonel, 
Chief,  Contracting  Division 


USAF 


c^fmnlea. 
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ABReadia-BJ _ Management  Survey 


NOTE;  The  survey  questions  used  for  this  research  are 
highlighted  in  bold  print. 


FROM;  AFIT/LSA  29  March  1993 

SUBJECT;  Survey  -  Organizational  Performance  Factors  in 
Operational  Contracting  Squadrons 

TO;  Survey  Recipient 

As  part  of  two  thesis  efforts  at  the  Air  Force  Institute  of 
Technology  (AFIT),  we  are  attempting  to  develop  improved 
management  tools  for  operational  contracting  squadrons.  To 
do  this,  the  attached  survey  attempts  to  gather  your 
opinions  and  judgments  about  factors  significant  to  a 
contracting  squadron's  performance.  Copies  of  the  survey 
are  being  sent  to  squadron  commanders,  deputy  base 
contracting  officers,  and  contracting  squadron  first 
sergeants  at  all  bases  within  the  CONUS. 

As  a  senior  manager,  you  are  in  a  unique  position  to  provide 
a  critical  body  of  information  necessary  for  these  thesis 
efforts. 

We  estimate  that  completion  of  the  survey  will  only  take 
about  twenty-five  minutes.  The  opinions  and  judgment  of 
experienced  personnel  such  as  you,  will  provide  a 
significant  contribution  to  the  success  of  these  studies. 

The  enclosed  survey  was  pilot  tested  by  a  sample  of 
contracting  squadron  personnel.  Based  upon  the  pilot  test, 
the  survey  was 

revised  in  order  to  obtain  all  necessary  data  while 
requiring  a  minimum  of  your  time. 

Please  mark  your  responses  directly  on  the  survey.  Mo 
coding  sheet  is  required,  ffhen  complete,  please  return  the 
completed  survey  in  the  enclosed  envelope. 

Due  to  deadlines  established  by  AFIT,  and  the  fact  that 
other  phases  of  both  research  projects  cannot  be  carried  out 
until  analysis  of  the  survey  data  is  complete,  we  would 
appreciate  if  you  complete  and  return  the  subject  survey  by 
3  May  1993. 
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Your  cooperation  in  this  matter  is  greatly  appreciated. 

Your  input  will  provide  great  insight  into  how  operational 
contracting  squadrons  can  improve  their  overall  performance. 


MARK  W.  FAHRENKAMP,  CAPT,  USAF 
AFIT  Graduate  Student 


MARK  P.  GARST,  CAPT,  USAF 
AFIT  Graduate  Student 


DOUGLAS  E.  JAMES,  CAPT,  USAF 
AFIT  Graduate  Student 


DENNIS  W,  GROSECLOSE,  ILT,  USAF 
AFIT  Graduate  Student 
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SURVEY 


PARTL  Background 

This  section  of  the  survey  obtains  information  about  your  background.  The 
information  requested  is  to  ensure  the  groups  you  belong  to  arc  accurately  represented, 
not  to  identify  you  as  an  individual.  Your  anonymity  will  be  maintained  throughout  this 
study. 

1.  Total  months  in  present  job  position. 


2.  Which  of  the  following  applies  to  you? 

1.  Civilian 

2.  Officer 

3.  Enlisted 

3.  How  many  years  of  contracting  experience  do  you  have? 


4.  What  major  command  are  you  presently  assigned  to? 

1.  Air  Mobility  Command 

2.  Ail  Combat  Command 

3.  Air  Training  Command 

4.  Air  Force  Material  Command 

5.  Other 

5.  How  impoitant  is  it  to  you  that  your  organization  achieve 
optimal  effectiveness? 

1.  Of  no  importance. 

2.  Of  slight  importance. 

3.  Moderately  important. 

4.  Faii'ly  important. 

5.  Extremely  important. 


102 


6.  What  was  the  last  overall  command  Inspector  General  (IG)  rating  of 
your  organization? 

1.  Outstanding 

2.  Excellent 

3.  Satisfactory 

4.  Marginal 

5.  Unsatisfactory 

6.  Does  not  apply. 

7.  Were  you  in  your  present  job  position  during  the  last  command  IG 
inspection? 

1.  Yes 

2.  No 

Please  indicate  your  opinions  on  the  following  two  statements; 

8.  The  current  command  IG  evaluation  system  satisfactorily 
measures  organizational  performance. 

1.  Strongly  disagree. 

2.  Disagree. 

3.  Neutral. 

4.  Agree. 

5.  Strongly  agree. 

9.  An  evaluation  system,  different  from  the  command  IG,  would  be 
useful  in  measuring  overall  organizational  performance. 

1.  Strongly  disagree. 

2.  Disagree. 

3.  Neutral. 

4.  Agree. 

5.  Strongly  agree. 
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PART  n.  ORGANIZATIONAL  PERFORMANCE  FACTOl  5 


This  portion  of  the  surv  ey  contains  factors  sometimes  used  to  measure 
performance  of  operational  contraaing  squadrons.  Please  indicate  the  importance  you 
would  assign  to  each  factor  by  circling  the  appropriate  number  on  the  scale  printed  to  the 
right  of  each  factor.  Scale  values  are  shown  below: 

Scale  of  Importance 

DehnitelyNot  Somewhat  Not  Somewhat  Definitely 

Important  Important  Neutral  Important  Important 


1 


PERFORMANCE  FACTOR 

10.  Number  of  open  purchase  requests. 

11.  Number  of  Small  Business  goals  achieved. 

12.  Last  IG  rating. 


SCAI.E  OF  IMPORTANCE 
1  2  3  4  5 
1  2  3  4  5 
1  2  3  4  5 


13.  Command  or  higher  contracting  awards  (i.e.  Best  Contracting  1  2  3  4  5 
Squadron  in  ACC,  Professionalism  in  Contracting,  etc.) 


14.  Outstanding  Unit  Citation. 

15.  Number  of  contracts  behind  schedule  (delinquent). 

16.  Total  number  of  delinquent  contractors. 

17.  Number  of  SF  129  packages  mailed. 

18.  Number  of  new  vendors  identified  and  loaded  in  BCAS. 

19.  Number  of  Reports  of  Discrepancy  (RODS). 

20.  Number  of  STEP  promotions. 

21.  Number  of  protests  received. 


1  2  3  4  5 
1  2  3  4  5 
1  2  3  4  5 
1  2  3  4  5 
1  2  3  4  5 
1  2  3  4  5 
1  2  3  4  5 
1  2  3  4  5 


Juab 
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Scale  of  Importance 


Definitely  Not 
Important 


Somewhat  Not 
'  Important 


Neutral 


Somewhat 

Important 


Definitely 

Important 


PERFORMANCE  FACTOR 

22.  Number  of  protests  successfully  defended. 

23.  Total  number  of  ratifications. 

24.  Percentage  of  competitive  actions. 

25.  Number  of  personnel  assigned  versus  authorized. 

26.  Total  number  of  contracting  actions. 

27.  Total  number  of  centralized  actions. 

28.  Total  number  of  decentralized  actions. 

29.  Total  dollars  awarded  competitively. 

30.  Number  of  interest  payments  paid  due  to  late  processing 
of  contractor  invoice. 

31.  Number  of  contracts  awarded  using  source  selection. 

32.  Number  of  undefinitized  actions. 

33.  P  umber  of  BPAs  administered. 

34.  ?  'umber  of  contracting  officer  warrants. 

35.  Number  of  solicitation  amendments  due  to  contracting 
personnel  error. 


SCALE  OF  IMPORTANCE 


1  2  3  4  5 
1  2  3  4  5 
1  2  3  4  5 
1  2  3  4  5 
1  2  3  4  5 
1  2  3  4  5 
1  2  3  4  5 
1  2  3  4  5 
1  2  3  4  5 

1  2  3  4  5 
1  2  3  4  5 
1  2  3  4  5 
1  2  3  4  5 
1  2  3  4  5 


36.  Number  of  customer  education  classes/training  provided. 


1  2  3  4  5 


Scale  of  Importance 


Definitely  Not 
Imrportant- 


Somewhat  Not 
Important 


Neutral 


Somewhat 

Important 

_ I _ 


Definitely 

Important 


PERFORMANCE  FACTOR 


SCALE  OF  IMPORTANCE 


37.  Number  of  man-hours  expended  to  support  deployments,  1  2  3  4  5 

contingency  contracting  activities,  and  mobility  exercises. 

38.  The  ratio  of  formal  training  quotas  received  divided  1  2  3  4  5 

by  the  number  utilized. 

39.  The  number  of  in-house  proficiency  training  performed.  1  2  3  4  5 

40.  Number  of  squadron  Operating  Instructions.  1  2  3  4  5 

41.  Number  of  self  inspections  conducted.  1  2  3  4  5 

42.  Number  of  Value  Engineering  Change  Proposals  received.  1  2  3  4  5 

43.  Numberof  Value  Engineering  Change  Proposals  approved.  1  2  3  4  5 

44.  Percent  of  the  time  BCAS  is  available  to  users.  1  2  3  4  S 

45.  Number  of  Purchase  Requests  over  90  days  old.  1  2  3  4  5 

46.  Number  of  Purchase  Requests  over  120  days  old.  1  2  3  4  5 

47.  Number  of  vendor  follow-ups  due  to  late  delivery.  1  2  3  4  5 

48.  Number  of  personnel  qualified  for  at  least  APDP  level  I  certification,  1  2  3  4  5 


49.  Number  of  walk-through  purchase  requests. 

50.  The  number  of  administrators  and  buyers  (excluding 
.  management  and  procurement  clerks). 


1  2  3  4  5 
1  2  3  4  5 


5 1 .  The  number  of  clerical  support  personnel. 


1  2  3  4  5 
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Scale  of  ImporUnce 


Definitely  Not 

Somewhat  Not 

Somewhat 

« 

Definitely 

Important 

Important 

Neutral 

Important 

Important 

1 


3 


4 


5 


PERFOIRMANCE  FACTOR 


SCALE  OF  IMPORTANCE 


52.  The  number  of  management  personnel  (includes  front  office  1  2  3  4  5 

branch  or  flight  chiefs,  and  Executive  Officer/NCO). 

53.  The  number  of  assigned  civilian  personnel.  1  2  3  4  5 

54.  The  number  of  assigned  officer  personnel.  1  2  3  4  5 

55.  The  number  of  assigned  enlisted  personnel.  1  2  3  4  5 

56.  The  total  office  experience  level.  1  2  3  4  5 


57.  The  average  office  experience  level  (total 
experience  divided  by  total  number  of  personnel). 

58.  The  total  office  experience  level  vvithout  procurement 
clerk  or  administrative  support. 

59.  The  average  office  experience  level  without 
procurement  clerk  or  administrative  support, 

60.  The  total  office  operating  budget  (money  you  have  to 
operate  the  office). 

61.  The  total  number  of  tiixe  items  received  priority  1*8. 


1  2  3  4  5 

1  2  3  4  5 

1  2  3  4  5 

1  2  3  4  5 

1  2  3  4  5 


62.  The  total  number  of  line  items  received  (all  priorities).  1  2  3  4  5 


63.  The  number  of  different  customer  organizations  served  by  1  2  3  4  5 

our  office. 


64.  The  total  dollars  awarded. 


1  2  3  4  5 
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1 


Scale  of  Importance 


Definitely  Not 
Important 


Somewhat  Not 
Important 


Neutral 


Somewhat 

Important 


Definitely 

Important 


PERFORMANCE  FACTOR 


65.  The  total  number  of  modifications  executed. 


66.  The  priority  1-3  CALT. 

67.  The  priority  4-8  CALT. 


68.  The  priority  9-15  CALT. 

69.  The  overall  (average)  CALT. 

70.  The  total  number  of  Non- Appropriated  Fund  actions. 

71.  The  total  Non-Appropriated  Fund  dollars  awarded. 

72.  The  total  number  of  Section  8A  actions. 

73.  The  total  Section  8A  dollars  awarded. 

74.  The  total  number  of  line  items  awarded. 

75.  The  total  number  of  centralized  line  items  awarded. 

76.  The  total  number  of  decentralized  line  items  awarded. 

77.  The  total  centralized  dollars  awarded. 

78.  The  total  decentralized  dollars  awarded. 

79.  The  ratio  of  centralized  actions  divided  by  centralized 
line  items  (measure  of  combining  multiple  line  items 
into  one  action). 


SCALE  OF  IMPORTANCE 
1  2  3  4  5 
1  2  3  4  5 
1  2  3  4  5 
1  2  3  4  5 
1  2  3  4  5 
1  2  3  4  5 
1  2  3  4  5 
1  2  3  4  5 
1  2  3  4  5 


1  2  3  4  5 
1  2  3  4  5 
1  2  3  4  5 
1  2  3  4  5 
1  2  3  4  5 
1  2  3  4  5 
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Scale  of  Importance 


Definitely  Not 

Somewhat  Not 

Somewhat 

Definitely 

Important 

Important 

Neutral 

Important 

Important 

1 


2  3  4  5 


PERFORMANCE  FACTOR 


SCALE  OF  IMPORTANCE 


80.  The  ratio  of  decentralized  dollars  awarded  divided  by 

total  dollars  awarded  (measure  of  decentralizing  workload). 

81 .  The  ratio  of  decentralized  line  items  awarded  divided  by 

total  line  items  awarded  (measure  of  decentralizing  workload). 


The  following  ratio  gives  a  percentage  of 
in-house  awards  which  required  modification  (a  low  percentage 
is  good). 


82.  The  ratio  of  total  modifications  executed  divided  by 
total  centralized  actions  awarded. 


1  2  3  4  5 

1  2  3  4  5 

1  2  3  4  5 


83.  The  total  set-aside  actions  awarded. 


1  2  3  4  5 


84.  The  total  number  of  large  business  actions  awarded  that  1  2  3  4  5 

were  available  for  small  business  (a  measure  of 

dissolving  set-asides). 

85.  The  total  large  business  dollars  awarded  that  were  1  2  3  4  5 

available  for  small  business. 

) 

86.  The  ratio  of  small  business  dollars  awarded  divided  1  2  3  4  5 

by  total  dollan  available  for  small  business. 

87.  The  percent  competitive  dollars  measure  (competitive  1  2  3  4  5 

dollars  divided  by  total  dollars  awarded). 

88.  The  total  number  of  active  contracts  administered.  1  2  3  4  5 


89.  The  total  number  of  active  A&E  and  construction  contracts  1  2  3  4  5 
administered. 
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Scale  of  Importance 


Definitely  Not  Somewhat  Not  Somewhat  Definitely 

Important  Important  Neutral  Important  Important 


1  2 

3 

4  5 

f 

PERFORMANCE  FACTOR 

SCALE  OF  IMPORTANCE 

90.  The  total  number  of  active  service  contracts  administered.  1  2  3  4  5 

91.  The  total  number  of  active  commodities  contracts  1  2  3  4  S 

administered. 

Note  for  question  92:  Priced  actions  represent  actions  that 
were  purchased  fi'om  such  pre-priced  instruments  as  calls  against 
blanket  delivery  orders  or  blanket  purchase  agreements  and 
delivery  orders  against  pre-priced  contracts.  Unpriced  actions 
are  those  actions  which  required  pricing  action  prior  to  award. 

This  includes  items  such  as  purchase  orders,  centralized  BP  As, 
imprest  fund  actions  and  contracts. 

92.  The  ratio  of  priced  actions  awarded  divided  by  1  2  3  4  5 

unpriced  actions. 

PART  UI;  ADDITIONAL  PERFORMANCE  FACTORS 

Please  add  any  additional  factors  you  have  used  to  evaluate  contracting 
performance  and  indicate  the  importance  of  each. 


93. 

94. 

95. 

96. 

97. 


1  2  3  4  5 
1  2  3  4  5 
1  2  3  4  5 
1  2  3  4  5 
1  2  3  4  5 
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PART  IV.  OVERALL  PERFORMANCE 


« 

Using  the  factors  listed  above,  including  the  factors  you  may  have  added  in 
questions  93-97,  select  the  five  you  feel  are  the  most  important  in  defining  organizational 
performance  within  an  operational  contracting  squadron.  Indicate  your  ranking  of  these 
five  factors  by  inserting  their  question  number  in  the  blanks  below. 


FIRST  in  SECOND  in  THIRD  in  FOURTH  in  FIFTH  in 
Importance  Importance  Importance  Importance  Importance 


PART  V.  OVERALL  PERFORMANCE  OF  YOUR 
SQUADRON/ORGANIZATION 

Based  upon  the  following  scale,  how  would  you  rank  the  overall  performance  of 
the  contracting  squadron/organization  you  are  assigned.  This  question  is  based  solely  on 
your  belief,  not  the  last  IG  rating  or  command  evaluation.  Please  circle  the  appropriate 
level  of  performance. 


Unsatisfactory  Marginal  Satisfactory  Excellent 


Outstanding 


PART  VL  DESIRABLE  CHARACTERISTICS  OF  A  PERFORMANCE 
EVALUATION  AND  FEEDBACK  SYSTEM 


This  pul  of  the  survey  is  designed  to  obtain  managers  views  and  desires 
conceroing  a  pertbrmance  evaluation  and  feedback  system  for  operational  contracting.  In 
addition,  this  section  contains  questions  concerning  current  methods  of  evaluating  and 
providing  feedback  to  the  operational  contracting  squadron.  Please  indicate  the 
importance  you  would  assign  to  the  statements  listed  below.  Continue  using  the  following 
scale. 


Definitely  Not 
Inqjortant 


Scale  of  Importance 

Somewhat  Not  Somewhat 

In^mrtant  Neutral  Important 


Definitely 

Important 


1 


.5345 


PERFORMANCE  FACTOR 


SCALE  OF  IMPORTANCE 


98.  The  current  IG  method  of  performance  evaluation  provides 
feedback  to  help  improve  performance  which  is: 

99.  Current  command  and  Air  Force  awards  provide  feedback 
concerning  perfoimance  which  is. 

100.  Improving  operational  contraaing  squadron  produaivity 
and  efficiency  is  a  goal  which  is: 

101 .  A  contracting  evaluation  system  which  simultaneously 
evaluates  several  inputs  and  outputs  to  the  process 

(as  opposed  to  single  measures  such  as  CALT),  is  a  tool 
which  is: 

102.  A  contracting  evaluation  system  which  compares  all 
operational  contracting  squadrons  while  taking  into 
account  differences  in  squadron  characteristics  (such  as 
manning,  experience,  and  workload)  is  a  tool  which  is: 

103.  A  contracting  '^'aluation  system  which  does  not  rely 
solely  upon  measures  which  the  chain  of  command 
(LG,  Wing  CC)  find  important,  is  a  tool  which  is 


1  2  3  4  5 

1  2  3  4  5 

1  2  3  4  5 

1  2  3  4  5 

1  2  3  4  5 

1  2  3  4  5 
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Scale  of  Importance 


Definitely  Not 

Somewhat  Not 

Somewhat 

Definitely 

Important 

[ 

Important 

1 

Neutral 

1 

Important 

1 

Important 

1 

1 

2 

3 

4 

5 

PERFORMANCE  FACTOR  SCALE  OF  IMPORTANCE 

104.  An  evaluation  system  which  compares  squadrons  relative  1  2  3  4  5 

to  other  contracting  squadrons,  is  a  tool  which  is: 

105.  A  contracting  evaluation  system  which  compares  contracting  1  2  3  4  5 

squadrons  to  the  best  perfoimers  as  opposed  to  die 

average,  is  a  tool  which  is: 

106.  A  contracting  evaluation  system  which  provides  managers  1  2  3  4  5 

with.timely  feedback,  including  exact  data  on  resource 

utilization  and  relative  efficiencies  compared  with 
other  squadrons,  is  a  tool  which  is: 

PART  Va  ADDITIONAL  COMMENTS  ON  CONTRACTING  PERFORMANCE 

Using  the  space  provided  below,  please  feel  free  to  add  any  additional  comments 
you  may  have  concerning  th.  performance  of  operational  contracting  squadrons. 
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Appendix  C; _ Surveyed  Bases 


Andrews  AFB 

Mcconnel  AFB 

Minot  AFB 

Moimtain  Home  AFB 

Eglin  AFB 

Hanscom  AFB 

Hill  AFB 

Hurlburt  Field 

Barksdale  AFB 

Bergstrom  AFB 

Carswell  AFB 

Davis-Monthan  AFB 

Dyess  AFB 

F.E.  Warren  AFB 

Ellsworth  AFB 

Dover  AFB 

Coluiabus  AFS 

Castle  AFB 

Cannon  AFB 

Beale  AFB 

Altus  AFB 

Lowry  AFB 

Little  Rock  AFB 

L£mgley  AFB* 

KI  Sawyer  AFB 

Holloman  kFh 

Griffiss  AFB 

Lackland  AFB 

Goodfellow  AFB 

Loring  AFB 

Laughlin  AFB 

Keesler  AFB 

Grissom  AFB 

Grand  Forks  AFB 

Fairchild  AFB 

Luke  AFB 

Malmstrou  AFB 

MacDill  AFB 

March  AFB 

Maxwell  AFB 

McGuire  AFB 

McCord  AFB 

Moody  AFB 

Morton  AFB 

Offutt  AFB* 

Falcon  AFS 

Charleston  AFB 

Tyndall  AFB 

Sheppard  AFB 

Seymour  Johnson  AFB 

Reese  AFB 

Randolph  AFB 

Plattsburgh  AFB 

Patrick  AFB 

Travif  AFB 

Shaw  AFB 

Scott  AFB 

Langley  AFB* 

Los  Angeles  AFB 

McClellan  AFB 

Nellis  AFB 

Newark  AFS 

Pope  AFB 

Peterson  AFB 

Onizuka  AFB 

Wurtsmith  AFB 

Whiteman  AFB 

Williams  AFB 

Vandenberg  AFP 

Wright  Patterson  AFB 

Edwards  AFB 

Brooks  AFB 

Kelly  AFB 

Kirtland  AFB 

Bolling  AFB 

Arnold  AFS 

Robbins  AFB 

Scott  AFB 

Shaw  AFB 

Tinker  AFB 

USAF  Academy 

Gunter  AFB 

Vance  AFB 

This  represents  a  base  with  two  contracting  squadrons. 
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Performance  ACC  AMC  AFMC  ATC  other  Total  Staff 
factor  Avg 


of  4.69  4.72  4.62  4.74  4.73  4.699  4,54 

urcnase 
equests 
ver  120 


Number  of  4.57  4,52  4.54  4.65  4.2  4.536  4.46 

purchase 

requests 

over  90  days 

(45) 


4.55  4.32  4.46  4.39  4.484  4.31 


4.451  4.31 


Number  of  4.39  4.52  4.77  4.65"  4.6  4.503 

contracts 
behind 
schedule 
(15) 


Number  of  4.52  4.48  4.39 

man-'hours 

expended  to 

support 

deployments , 

etc.  (37) 


Number  of  4.47  4.44 

amendments 

due  to 

contracting 

error  (35) 


Total  number  4.43  4.16  4.69  4.65  3.73 

of  line 
items 

received  all 
priu  rities 
(62) 


Total  number  4.32  4.24  4.38  4.61  4. 

of  active 
service 
contracts 
90) 


4.22  I  4.33  I  4.418  I  4.23 


4.373  4.23 


4.366 


Total  number  4.3l'4.2  4.38  4.43  4.4  4.359  4.15 

of  active, 
contracts 
(88) 


Performance  ACC  AI4C  I AFMC  ATC  Other  Total  Staff 
factor  Avg 


Hiuober  of  4.32  4.24  4.38  4.30  4.4  4.317 

customer 
education 
classes 
rovided  (36^ 


4.18  4.36  4.31  4.52  4.53  4.307  3.77 


4.307  1 4.31 


Total  number  4.31  4  4.77  4.57 

of  line  items 
received 
priority  1-8 
(61) 


Total  number  4.25  4.16  4.46  4.52  4.07 

of  active  A&E 
and 

construction 

contracts 

(89) 


Total  number  4.25  4.08  4.38  4.57  3.93 

of 

commodities 

contracts 

(91) 


Number  of  in-  4.21  4.08  4.31  4.35  4.27 

house 

proficiency 
training 
sessions  (39) 


4.29  4.0e  4.23  4.17  3.87 

pen  purchase 
equests  (10) 


4,275 


4.248  3.85 


4.222 


Total  number  4.13  4.32  4.31  4.17  4.2 

of  delinquent 
contractors 
(16) 


4.190  I  3.85 


4.190  3.92 


AMC  I  AFMC  I  ATC  I  Other 


Staff 


Performance 

factor 


Number  of  4.17  4.12  4.54  4.09  4.13  4.176,  3.85 

interest 
pa^nnents  paid 
due  to  late 
processing  of 
contractor 
invoice  r30> 


4.13  4.08  4.31  4.43  3.93  4.163  3.54 


4.09  I  4.53  I  4.144  i  4.08 


Total  number  4.12  4.32  4.46  4.04  3.8 

of 

modifications 
J65) _ 


Number  of  4.05  4.24  4.62  4.04 

vendor 
follow-ups 
due  to  late 

delivery  (47) _ !_ 

Total  dollars  4.03  4.08  3.77  4.22  4.2 

awarded 
competitively 
(29) 


Number  of 

different 

customer 

organizations 

served  by 

contracting 

(63) 


Total  number  4.13  3.84  3.85 

of 

ratifications 
(23) 


4.03  I  3.8  I  4.31  I  4.13  I  3  87 


4.137 


4.124  I  4.15 


4.059  4 


3.92  I  4.31  I  4.48  I  3.53  I  4.039  I  3.54 


3.93  I  4.026  I  3.85 


4.013 


4.04  4.08  4 


Total  number 
of 

centralized 
actions  (27) 


3.97  4.04  4.15 


4,39  3.2  I  4.013  I  4.23 


4.007  I  3.38 


117 


Performance  ACC 
factor 


Total  number 
of 

centralized 
line  items 
awarded  (75) 


AFMC  I  ATC  I  Other 


4.04  4.15  4.26 


Staff 


3.980  3.62 


3.980  4.08 


3.92  3.96  3.69  4.13  4.07  3.954  I  3.62 


3.95  3.76  4.31  4.09  3.6  3.935  3.85 


3.81  4.2  4.46  I  4.13 


3.84  3.84  4.15  4.13  3.6 


Total  number  3.87  4.04  3.54  4.22  3.93 

of  large 
business 
actions 
awarded  that 
were 

available  for 
small 

business  (84) 


N\uaber  of 
self 

inspections 

conducted 

(41) 


Total  large  3,90  3.88  3.46  4,22  3.67 

business 
dollars 
awarded  that 
were 

available  for 
small 

business  (85) 


Number  of 
Reports  of 
Discrepancy 
19 


3.928  13.31 


3.889  3.85 


3.882  3.31 


3.869  3.38 
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Performance 

factor 

ACC 

AMC 

AFMC 

ATC 

Other 

Total 

Avq 

Staff 

E^BSluaii 

3.86 

3.92 

3.92 

3.57 

3.73 

3.817 

3.62 

3.83 

3.54 

4.15 

3.96 

' 

3.67 

3.814 

3.38 

Total  Section 
8A  dollars 
awarded  (73) 

3.79 

3.76 

3.69 

3.96 

3.53 

3.7*78^ 

3.62 

Total  set- 
aside  actions 
awarded  (83) 

3.79 

3.84 

3.77 

4.04 

3.13 

3.771 

3.38 

Ratio  of 
small 
business 
dollars 
awarded 
divided  by 
total  dollars 
available  for 
small 

business  (86) 

3.81 

3.88 

3.69 

3.74 

3.53 

3.771 

3.23 

^iSfti^\yirpip  y^T- T>  w 

3.71 

4 

3.69 

3.93 

3.47 

3.768 

3.23 

Total 

centralized 
dollars 
awarded  (77) 

3.77 

3.92 

3.69 

4 

3.2 

3.765 

3.31 

Number  of 
small 
business 
goals 

achieved  (11) 

3.78 

3.56 

3.54 

4 

3.8 

3.758 

3.77 

3.78 

3.68 

3.62 

4.13 

3.33 

3.758 

3.62 

3.  5 

3.8 

3.69 

3.87 

3.6 

3.758 

3.69 

Number  of 

BPAs 

administered 

(33) 

3.58 

3.72 

3.92 

3.70 

3,47 

3.641 

3.31 

Total  nuxnber 
of 

decentralized 
line  items 
awarded  (76) 

3.58 

3.88 

3.62 

3.91 

2.8 

3.608 

3.31 
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factor 


Outstanding  3.66 
Unit  Citation 
(14) 


Number  of 
contracts 
awarded  using 
source 
selection 
31 


Number  of 
STEP 

promotions 
20 


Total  number 
of  Non- 
Appropriated 
Fund  actions 
70 


3.51  3.36  3 


3.06 


Number  of 
Value 

Engineering 


Proposals 
aoD roved  (43 


identified 
and  loaded  in 
BCAS  (18 


Number  of 
squadron 
Operating 
Instructions 
40 


2.86 


AFMC 

ATC 

Other 

3.62 

3.39 

3.13 

3.31 

3.78 

2.73 

4 

! 

3.30 

3.47 

3.92 

3.17 

3.07 

3 

3.22 

3 

3.54 

2.96 

1 

3 

3.23 

2.96 

2.53 

3.23 

3 

2.73 

1 

3.08 

2.78 

2.73 

3.23 

2,65 

j 

2.47 

3 

2.87 

2.27 

Staff 


8 


3.350  3 


3.105  2.77 


2.958  2.92 


2.797  1 2.69 


Question  # 


Mean 


Variance 


Std  Dev 


46  1  451  15  1  44  1  37  1  35  ,  62 


719  694  689  686  681  676  669 


4.699346  4.535943  4.503268  4.48366  4.45096  4.418301  4.372549 


0.290592  0.381923  0.580581  0.396113  0.565015  0.573873  0.630031 


0.539066  0.617999  0.761959  0.629375  0.751675  0.757544  0.793745 


Numerator 

0.699346 

Denominator 

0.043581 

Test  Stat 

16.0471 

0.503268 

0.48366 

0.45098 

0.418301 

0.372549 

0.061601 

0.050882 

0.060769 

0.061244 

0.06417 

8.169838 

9.505535 

7.421184 

6.830087 

5.805614 

One  tail 


99%test 


95%test 


90%test 


99%  test 


95%  test 


90%  test 


2.326  2.326  2.326  1  2.326  1  2.326  2.326  2.326 


1.6451  1.6451  1.645]  1.645  1.645  1.645 1  1.645 


1.2821  1.2821  1.2821  1.2821  1.2821  1.2821  1.282 


REJECT!  REJECT  REJECT  REJECT]  REJECT 


REJECT  REJECT  REJECT  REJECT  REJECT  REJECT]  REJECT 


REJECT  REJECT  REJECT  REJECT  REJECT  REJECT  REJECT 


llrlJM.BEiaHsai 


Question  # 

47 

29 

63 

23n 

66 

74 

27 

Total 

631 

621 

618 

616 

614 

614 

613 

Mean 

4.124183 

4.058824 

4.039216 

4.026144 

4.013072 

4.013072 

4.006536 

Variance 

0.701582 

0.832043 

0.880031 

0.94668 

1.276144 

0.973512 

0.677589 

Std  Dev 

0.837805 

0.912164 

0.9381 

0.972975 

1.129685 

0.986667 

0.823158 

Numerator 

0.124183 

0.058824 

0.039216 

0.026144 

0.013072 

0.013072 

0.006536 

Denominator 

0.067716 

0.073744 

0.075841 

0.07866 

0.091328 

0.079767 

0.066548 

Test  Stat 

1.83387 

0.797871 

0.517079 

0.332363 

0.143131 

0.163875 

0.098214 

One  tail 

9~9%test 

2.326 

2.326 

2.326 

2.326 

2.326 

2.326 

95%test 

1.643 

1.645 

1.645 

1.645 

1  645 

1.645 

1.645 

90%test 

1.282 

1.282 

1.282 

i.282 

1.282 

1.282 

1.282 

99%  test 

FTR 

FTR 

FTR 

FTR 

FTR 

FTR 

FTR 

95%  test 

REJECT 

FTR 

FTR 

FTR 

FTR 

FTR 

FTR 

90%  test 

REJECT 

FTR 

FTR 

■■023 

FTR 

FTR 

FTR 
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Quetstion  # 

75 

40 

87 

67 

84 

41 

85 

Totai 

eo9 

60S 

605 

602 

601 

591 

504 

Mean 

3.080302 

3.080263 

3.954248 

3.034841 

3.928105 

3.888889 

3  882353 

Variance 

0.014087 

0.950873 

1.057104 

1.2457 

1.027692 

0.994152 

0.986068 

StdDev 

0.979731 

1.028155 

1.116109 

1.013751 

0.997072 

0.99301 

Numerator 

•0.01031 

-0.01974 

-0.04575 

-0.06536 

-0.0719 

-0.11111 

-0.11765 

0.077204 

0.070207 

0.083121 

0.000232 

0.081957 

0.080808 

0.08028 

TestStat 

-0.25308 

-0.24918 

•0.55042 

-0.72435 

■0.87723 

-1.3784 

•1.46546 

One  tail 

oe%test 

2.326 

2.326 

2.326 

2.326 

2.326 

2.326 

2.326 

05%te8t 

1.645 

1.645 

1.645 

1.645 

1.645 

1.645 

1.645 

90%test 

1.282 

1.282 

1.282 

1.282 

1.282 

1.282 

1.282 

00%  test 

FTR 

FTR 

FTR 

FTR 

FTR 

FTR 

FTR 

95%  test 

FTR 

FTR 

FTR 

FTR 

FTR 

FTR 

FTR 

90%  test 

FTR 

FTR 

FTR 

FTR 

FTR 

FTR 

FTR 

Question  # 

19 

69 

68 

21 

32 

73 

83 

Total 

592 

592 

58^ 

584 

583.5 

578 

577 

Mean 

3.869281 

3.889281 

3.830065 

3.816993 

3.813725 

3.777778 

3.771242 

Variance 

0.877537 

1.29859 

1.328195 

1.032078 

0.999613 

0.910319 

1 .032852 

Std  Dev 

0.936769 

1.139557 

1.151606 

1.015912 

0.999806 

0.954388 

1.016293 

Numerator 

-0.13072 

-0.13072 

-0.16993 

-0.18301 

-0.18627 

-0.22222 

-0.22876 

Denominator 

0.075733 

0.092128 

0.093102 

0.082132 

0.08083 

0.077156 

0.082162 

Test  Stat 

-1.72604 

-1.41889 

-1.82526 

wi'M'mu 

-2.30453 

•2.88016 

-2.78422 

One  tail 

99%test 

2.326 

2.326 

2.326 

2.326 

2.326 

■EEEl 

2.326 

95%test 

1.645 

1.645 

1.645 

1.845 

1.645 

1.645 

1.645 

90%test 

1.282 

1.282 

1.282 

1.282 

1.282 

1.282 

1.282 

99%  test 

FTR 

FTR 

FTR 

FTR 

■Eia 

FTR 

FTR 

95%  test 

FTR 

FTR 

FTR 

FTR 

FTR 

FTR 

FTR 

90%  test 

FTR 

FTR 

FTR 

FTR 

FTR 

FTR 

FTR 

Question  # 

76 

14 

78 

31 

13 

12 

20 

Total 

552 

534 

532 

525 

521.5 

512.5 

475 

Mean 

3.607843 

3.490196 

3.477124 

3.431373 

3.408497 

3.349673 

3.104575 

Variance 

1.016254 

1.409443 

1.040592 

1.062693 

1.537625 

1.602253 

1.75215 

Std  Oev 

1.008094 

1.1872 

1.020094 

1.03087 

1.24001 

1.265801 

1.323688 

Numerator 

>0.39216 

-0.5088 

-0.52288 

-0.56863 

-0.5915 

-0.65033 

-0.89542 

Denominator 

0.0815 

0.095979 

0.08247 

0.083341 

0.100240 

0.102334 

0.107014 

Test  Stat 

-4.81177 

-5.3116 

-6.34022 

-6.82291 

-5.90035 

-6.35494 

-8.36737 

One  tail 

99%test 

2.326 

2.326 

2.326 

-2.326 

-2.326 

-2.326 

-2.326 

95%test 

1.645 

1.645 

1.645 

-1.645 

-1.645 

-1.645 

-1.645 

90%test 

1.282 

1.282 

1.282 

-1.282 

-1.282 

-1.282 

-1.282 

99%  test 

FTR 

FTR 

FTR 

FTR 

FTR 

FTR 

FTR 

95%  test 

FTR 

FTP 

FTR 

FTR 

mam 

FTR 

FTR 

90%  test 

FTR 

FTR 

FTR 

FTR 

FTR 

FTR 
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Question  i* 

70 

71 

43 

18 

40 

42 

17 

Total 

452.5 

450.5 

440 

432 

428 

426 

398 

Mean 

2.957516 

2.944444 

2.875817 

2.797386 

2.784314 

2.601307 

Variance 

0.907723 

0.959064 

1.03053 

1.291022 

,1.175783 

0.986068 

1.136051 

Std  Dev 

0.952745 

0.979318 

1.01515 

1.136231 

1.084335 

0.99301 

1.065857 

Numerator 


Denominator 


Tost  Stat 


One  tail 


99%test 


9S%tust 


90%test 


99%  test 


95%  test 


90%  test 


•1.04248  -1.05556  -1.12418  -1.17647  -1.20261  -1.21569 


0.077025  0.079173  0.08207  0.091859  0.087863  0.08028 


-13.53441  -13.3322  -13.6978  -12.80741  -13.71881  -15.1431 


-2.326  -2.31  ?  -2.326  -2.326 


-1.645 


-1.282 


FTR 


FTR 


FFR 


-2.326 


-1.645 


-1.282 


FTR 


FTR 


FTR 


-2.326 


-1.645 


•1.282 


FTR 


FTR 


FTR 


L3 


V 

App.gndjls  F;  Siqnifjgant  Fact.grs -N<?t 
Mgftgurfid  l2^.g.C&g 


Question  Performance  Factor 
Number 

46  Number  of  purchase  requests  over  120  days 

45  Number  of  purchase  requests  over  90  days 

44  Percent  of  time  BCAS  is  available  to  users 

37  Number  of  man-hours  expended  to  support 

deployments ,  etc . 

35  Nu^er  of  amendments  due  to  contracting  error 

36  Number  of  customer  education  classes  provided 

22  Number  of  protests  successfully  defended 

39  Number  of  in-house  proficiency  training  sessions 

10  Number  of  open  purchase  requests 

16  Total  number  of  delinquent  contractors 

30  Number  of  interest  payments  paid  due  to  late 

processing  of  contractor  invoice 

47  Number  of  vendor  follow-ups  due  to  late  delivery 

N/A  Customer  support  and  satisfaction 

*  based  on  input  from  write-in  response 
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